Preserving the Past, Enabling the Future: Assessing the European Policy on Access to Archives in the Digital Age

3.1 Digital Documents

According to the survey results on the current state of digitization, the majority of archives have a positive view of its impact on document accessibility. The primary reason for this positive assessment, cited by 73 % of archives, is the ability to access documents through the Internet. Digitized documents were also praised for their convenience, with 66 % of archives noting the advantage of eliminating the need for physical retrieval and transfer; furthermore, 61 % of respondents reported that digitization helps reduce the workload of archive staff. However, 55 % of archives agree that accessibility has only seen partial improvement thus far, as only a small portion of their holdings have been digitized or made remotely accessible.

Among the archives surveyed (42 in total, including 32 national and 10 regional/local archives), the proportion of born-digital text documents in their collections remains relatively low. Specifically, 86 % of archives reported that less than 5 % of their holdings consist of born-digital text documents, with an additional 9 % of archives reporting that the share of born-digital text documents ranges between 5 and 15 %. However, one national and one municipal archive reported that more than 50 % of their holdings are already born-digital.

Archives have also made significant efforts to digitize their analogue text documents (see Table 2). Among the responding archives, 56 % (18 national and five regional/local) have digitized less than 5 % of their analogue text documents, while another 34 % (10 national and four regional/local) have digitized between 5 and 15 % of their text documents, with a single national archive (2 %) achieving a digitization rate of between 25 and 50 %. Additionally, two national archives and one municipal archive reported that they have already digitized more than 50 % of their text documents.

Born-digital AV (audio, video and film) documents constitute a relatively small portion of the archives’ collections, but their overall share is slightly larger than that of text documents. Out of the 40 responding archives (30 national, 10 regional and local), 27 (68 %) have less than 5 % of their AV documents in born-digital format. In eight other archives (23 %), the share of born-digital AV documents is between 5 and 15 %, while two national Archives reported that more than 50 % of their AV documents were digitally created.

Regarding the digitization of analogue AV holdings, the situation differs from that of text documents (see Table 2). Among the responding archives, 64 % have digitized up to 5 % of their analogue AV documents, while another 21 % have digitized between 5 and 15 % of their AV documents. One national archive has digitized between 15 and 25 % of their AV documents, with another archive falling within the same range, while two more archives have digitized between 25 and 50 % of their AV documents. The leading national archive has already digitized more than 50 % of their AV documents, while the two most advanced municipal archives have digitized between 25 and 50 % of their AV documents.

According to Figure 1, 81 % of archives prioritize the protection of vulnerable documents for digitization, followed by 79 % wanting to make documents more accessible to a wider range of users, often in response to user demands. Efficiency gains in archival management were mentioned by only 31 % of respondents, with regular preservation and migration of digital documents of primary importance to 26 % of archives. Only 31 % mentioned efficiency gains in archival management, while 26 % digitize documents in the course of regular preservation and migration activities.

Figure 1:

Priority setting for digitisation. Question: When defining priorities in digitizing archival documents, which is of primary importance in your institution? Source: Questionnaire for Archives, Question 13.3, variables 376–80.

The possibility to provide digital documents over the Internet has posed new challenges for archives and other cultural heritage institutions, including user involvement in tasks like indexing archival material (Benoit, III and Eveleigh 2019; Ridge 2014). This “participatory archives” approach allows user contributions by enriching archival holdings and fostering a collaborative community. The Blinken OSA Archivum for instance has implemented two interactive projects: the first, Yellow Star Houses,^[7] commemorates a forgotten chapter of the Holocaust in Budapest, offering an interactive website with a map and related documents, where users can contribute photos and personal memories; the second project, Parallel Archive, enables researchers to upload and share archival documents, facilitating collaboration and context enhancement.

The archives were asked if they involve users in describing and/or commenting on archival documents through crowdsourcing, with such user involvement practiced in 23 % of the National Archives and 33 % of the regional and local Archives. This includes tasks like transcribing and indexing civil records, handwritten documents, and providing content descriptions for photographs. Archives may rely on volunteers, including amateur genealogists, or collaborate with platforms like Wikimedia.

Regarding the risks associated with increased online accessibility of documents, data protection and copyright violations were the most commonly mentioned risks, with both mentioned by 40 % of participants. Risks related to national security and document forgery or modification were considered less relevant, with only 12 % of participants mentioning each of these risks.

3.2 Remote Access

In addition to digital documents, remote access to finding aids and documents through the Internet is the second important component of a digital archive. Among the surveyed archives, 93 % provide online access to catalogues and finding aids but, although this is a high percentage, the extent of digital cross-references in the catalogues should also be taken into account (Figure 2). In national archives, only 11 % of online catalogues provide direct links to individual digital documents, while 59 % offer only descriptions of the documents. In contrast, larger regional and local archives tend to have a higher percentage of catalogues linking to digital documents, however, this may not be representative of small and medium-sized archives in regions and cities. Finally, we also asked whether the archives offer artificial intelligence-based applications for finding documents, with as many as 20 % of the national Archives having such applications in use, with none of the regional or local archives surveyed responding positively to this question.

Figure 2:

Cross-references in online catalogues. Question: If your catalogues/finding aids are available online, are there digital cross-references? Source: Questionnaire for Archives, Question 14.1.1, variable 351–54.

Regarding online access to digital documents, the majority of archives (88 %) offer this service for free. However, there are restrictions in place. In nearly half of national archives (48 %) and the majority of regional and local archives (90 %), not all digital documents are accessible online; instead, only certain collections/series or selected samples are available for online access (Figure 3).

Figure 3:

Free online access to digital documents. Question: If you provide free access to digital/digitized documents online, which parts can be accessed free of charge? Source: Questionnaire for Archives, Question 14.2.1, variable 357.

3.3 Online Communication

Communication channels with archive staff are vital for accessibility. As digitization and remote access increase, new digital communication channels have emerged to facilitate user requests and support, which include professional consultation, reading room reservation, and on-demand digitization requests (see Figure 4). Nearly all archives, both national and regional/local, provide support through their institutional website (90 % of national archives/100 % of regional and local archives), direct email with the reference archivist (79 %/90 %), and social networks (76 %/80 %). However, real-time chat with the reference archivist remains less common, available in only 15 % of the responding national archives.

Figure 4:

Possibilities for online communication with archives. Question: What kind of online channels does your institution use for communicating with archival users? Source: Questionnaire for Archives, Question 14.3, variable 363–66.

4.1 Increasing Scope of Data

Already at the stage of appraisal and selection, archives face a dilemma: should they preserve everything due to cheaper storage space and the potential future interest of even seemingly irrelevant documents and data, or should they continue to assess the current and future value of all available data and preserve only the valuable ones? Some argue that with the growth of the global information infrastructure, all information can and will be stored forever and accessible anytime, anywhere (Bell and Gemmell 2009).

If you follow this though the question arises as to whether society really needs specialized institutions like archives and other memory institutions that collect, preserve, and make information accessible. In response to this, Székely (2017, 11–14) identifies six key arguments supporting the continued relevance of archival institutions in the digital age:

1. Institutional traditions and their embeddedness in the cultural fabric of society;

2. Ensuring the long-term functionality of documents and data;

3. The ability to preserve physical, non-digital copies;

4. The importance of preserving the historical and information technology context;

5. The ongoing task of migrating document formats; and finally,

6. The important role of archival institutions in taking institutional responsibility.

Experts generally do not consider “total recall” as an effective approach. Simply storing documents and information without proper processing, cataloguing, and metadata enrichment does not guarantee easy accessibility, an issue which has been challenging for some time, even in the pre-digital era, resulting in the development of procedures for targeted selection or deletion (Mayer-Schönberger 2011).

Contemporary archives typically collect digital data and documents from the recent past, making it challenging to place them in a historical context. It is estimated that up to 20 % of digital documents will be archived, compared to 5 % in analogue contexts (Convery 2014, 159; Moss and Gollins 2017, 7). However, there is a concern that the size of digital collections may overwhelm smaller archives, as they may lack the resources to properly catalogue and index them.

The volume of data raises the question of prioritizing materials based on their importance and value. For example, the Federal Archives in Germany prioritize digitizing holdings from the National Socialist era due to high user demand (Hänger, interview). Similarly, in the Netherlands, there is a focus on digitizing holdings related to colonial history (Jeurgens, interview). While prioritization is necessary to ensure completeness for important topics in the digital collection, there is a risk of marginalizing less prioritized topics, leading to digitally inaccessible related documents.

4.2 Data Protection and Copyright Issues

The interviewed experts emphasized that the European General Data Protection Regulation (GDPR)^[8] is a primary challenge and threat to archive accessibility to date. This perception is influenced by the practical considerations and the stance of the competent data protection authorities. In response to GDPR, archives have often chosen to restrict access to potentially sensitive documents entirely, fearing legal violations and penalties, despite the previous successful handling of sensitive data by researchers using the archives (Sipos, interview).^[9] Archives feel uncertainty due to varying interpretations of GDPR requirements across the EU, find existing guidelines (EAG 2018) too abstract for their daily practice, and struggle to reconcile the tensions between data protection laws, freedom of information laws, and archive legislation.

Data protection concerns have led to the closure of entire digital collections, as highlighted by Jaillant (2022, 420f.) in an interview study. For example, the email correspondence of novelist Ian McEwan was transferred to a private Archive in the US in 2014 and is not even listed in finding aids. Even researchers who do have access to such closed collections face difficulties in publishing their research due to the time-consuming process of obtaining consent or rights to use the information from individuals involved. As a result, scholars tend to focus on more accessible document collections when choosing their research topics, a problem which affects not only born-digital documents but also digital copies of paper documents that were previously accessible in reading rooms.

Even if documents are (digitally) sanitized, combining them may still lead to the identification of individuals, violating data protection rules. Artificial intelligence tools can also uncover hidden patterns in data, potentially exposing sensitive personal or business information.

In addition to data protection, archives hesitate to make digital collections accessible for reasons beyond data protection. Copyright issues, especially when authors cannot be located, pose significant challenges;^[10] even when authors are known, acquiring usage rights can be time-consuming, causing archives to refrain from providing access to relevant documents (Hänger, interview).

Finally, accessibility may be restricted not only due to data protection and copyright concerns, but also because there may be “records that are culturally sensitive, or have offensive language in it, or are simply unpleasant to look at” (Jaillant 2022, 425).

4.3 Preservation and Interoperability

More than 25 years ago, when the discussion about digital archives started to spread, experts expressed concern about the possibility of significant data loss. They warned about a potential “digital dark age” due to the deterioration of storage media like magnetic tapes and floppy disks, as well as the aging of computer hardware (Hedstrom 1997; Kuny 1997). Additionally, the obsolescence of software and data formats often posed a significant barrier to accessing born-digital documents (Jaillant 2022, 423).

The current challenge is no longer focused on creating computer-readable documents in preservation-friendly formats like PDF/A. Instead, it lies in capturing and providing content and data from outdated systems, including obsolete email systems. Additionally, there is the question of determining the “original” version of a document, like a database, and whether the authenticity relies on the original IT environment and its technical properties (Friedewald and Leimbach 2011).

Effective and efficient data capture relies on personnel with the necessary skills, however, there is a shortage of appropriate IT knowledge in this area (Fresa, Justrell, and Prandoni 2015, 191; Kim 2018; Hänger, interview). Furthermore, ensuring the traceability of authenticity for born-digital objects poses an additional obstacle (Poole 2015, 116).

4.4 Organization of Records

Organizing digital documents also poses challenges for archives. The traditional concept of respect des fonds, where archivists passively preserve documents, is challenged in the digital realm, as digital collections often lack structure and proper metadata, requiring a more active curatorial approach for effective collection management (Jaillant 2022, 427f.).

Preservation and accessibility requirements are often in conflict with each other, as storing documents in non-future-proof formats necessitates repeated migration to ensure future accessibility, while obsolete systems and fragile data carriers, especially in the audio-visual field, pose additional challenges. The use of coding-decoding algorithms (codecs) can also impact the preservation of artistic qualities in video recordings; to address these challenges, an international community, known as “No Time To Wait!”, is developing new open standards and technical tools.^[11]

Standardized metadata schemes and adherence to international standards are crucial for effective access to digital documents across Archives. Open standards like the Open Archival Information System (OAIS) Reference Model^[12] and the Metadata Encoding and Transmission Standard (METS)^[13] are recommended. Elsewhere, the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)^[14] enables the collection of metadata descriptions from various archives, enabling the creation of services that utilize metadata from multiple sources. Metadata repositories act as aggregated information sources, allowing users to search within the repository and redirecting them to the original archive for relevant results. Aggregator portals like Archives Portal Europe^[15] and European^[16] provide integrated search capabilities across cultural institutions and collections, following similar principles.

In theory, the significant time gap between the creation of records and their opening up could encourage archivists to make access to collections as user-friendly as possible, while having ample time to process and organize records should also allow for designing strategies to make non-sensitive parts of the collections available sooner. However, without a clear understanding of how a future digital archive should be structured and the potential new tasks it may have, the long periods of closure can result in a hesitant “wait-and-see” attitude among archivists (Jaillant 2022, 429f.).

Within this context, experts highlight the notable differences in internal procedures followed by Archives for preservation and accessibility purposes (Hänger and Fickers, interviews). Preservation of documents can often be automated through mass processing techniques (scanning, text recognition), but ensuring accessibility requires time-consuming and challenging manual indexing and cataloguing.

New concepts of descripting standards have been developed, such as the “Records in Context” (ICA 2021), which includes a conceptual model and an ontology, and aims at enabling archival description to better capture the complex relationships archival documents have with each other and their creators, holders, and subjects. Linked Open Data is another concept of the Semantic Web, offering a common space for sharing data from repositories, including archives of a certain kind (Bizer, Vidal, and Skaf-Molli 2018). These concepts and the resulting standards may modernize existing descripting standards in the future, as well as inter-organizational cooperation, resulting in semantic interoperability and better accessibility of data and document repositories.

4.5 Automation and the Use of Emerging Technologies

As in other fields, there are high expectations for what artificial intelligence and machine learning (AI/ML) can achieve in the archival context. This was illustrated by Colavizza et al. (2022), who reviewed six years of English-language literature in the intersection of AI and archives, and analyzed 53 relevant publications using the Records Continuum model. Elsewhere, the journal AI and Society devoted a thematic section on the topic, “AI use in Archives” (e.g., Stapleton and Jaillant 2022). In this literature as well as in our expert interviews, two aspects in particular were mentioned where AI can improve the accessibility of (digital) archives:

1. AI/ML can automate the creation of metadata, particularly for large unstructured collections, which involves capturing and organizing the content of documents to provide a rough classification (Colavizza et al. 2022). However, it is important to consider algorithmic bias, as AI/ML learns biases inherent in the data. For instance, if documents from the colonial period are classified, there is a risk of reproducing European colonial terminology and racist stereotypes (Luthra et al. 2023; Jeurgens, interview).

2. AI/ML algorithms can automatically detect sensitive materials and differentiate between problematic and unproblematic content. They can also aid in indexing and disclosing digital documents, as well as identifying images like faces or buildings (Decker 2022). However, given the probabilistic nature of detection, there is a possibility that problematic documents may not be identified. Consequently, additional checks are required to prevent or minimize risks before making documents public.

In addition to these application areas, there will be further applications in the field of automatic person-, object-, and location recognition, which will help researchers to search and interactively enrich large archival collections. New types of sorting, tagging, extraction, translation, text-to-speech, and speech-to-text conversion of archival documents according to criteria defined by researchers will bring new possibilities to the work of users of archives.

4.6 User Interfaces

Creating a suitable interface for digital collections is vital for document accessibility. This includes addressing archival requirements to safeguard against modification or deletion, while it is also important to conceal sensitive data based on the user’s location and authorization, whether in a reading room or online.

In addition to viewing the document itself, users should have access to its broader or narrower context if available, which facilitates “scalable reading” which is especially useful for professional users. It enables them to seamlessly switch (or zoom) from distance to close reading (Fickers, interview; Fickers, Tatarinov, and Van Der Heijden 2022).

4.7 Cooperation of Memory Preserving Institutions and Outreach Beyond Traditional User Groups

Our survey primarily focused on traditional public archival institutions, particularly national archives. In these archives, it is important (though expensive and time-consuming) to ensure interoperability at a higher level; the survey results indicate a growing preference for online access among archive users, which necessitates the presence and online availability of digitized or born-digital documents. If researchers only find a portion of the documents they seek, they may expect the first archival institution to redirect them to another institution with supplementary material on the research topic, redirection which should be specific to the direct continuation of the document series in question. Developing such interoperable systems and standards, and making entire digital collections available in this manner, requires collaboration among institutions, however, these institutions often compete for resources and professional prestige, even if the benefits of these developments are realized elsewhere or contribute to improving user access in general. In the era of new types of archives, such as community archives or post-custodial archives, this collaboration becomes even more necessary.

New services that enhance accessibility to archival collections are attracting more users, but the demand is not primarily from traditional user groups. Instead, it comes from individuals with limited prior experience with archives who have new expectations. It is crucial to create appropriate accessibility methods for these users, which may differ from those for professional users, and experts hope that archivists, librarians, and museum curators will explore ways to generate increased interest in their collections and specific topics in the future (Sipos, interview). While exhibitions of original artifacts like handwritten manuscripts have been successful in capturing interest, digital documents lack the same materiality and aura. Memory institutions must develop new approaches to provide broader access to their collections for the public.