Coffee Chat Recap with Library and Archives Canada’s Tom Smyth

By Susan Paterson, SAA Web Archiving Section Vice-Chair and liaison librarian at Koerner Library at the University of British Columbia Vancouver.

The Web Archiving Section hosted another successful Coffee Chat which featured Tom Smyth, Program Manager of the Web and Social Media Preservation Program within the Digital Preservation Division at Library and Archives Canada (LAC). Tom discussed the evolution of the program which began in December 2005. The collection exceeds 100 TB of content and includes over 2.64 billion assets from wide ranging collections such as Government of Canada websites, thematic and rapid response collections. 

At the start of the LAC’s National Web Archive Program, collection activities focused on the Government of Canada domain, provincial and federal elections, Canada’s experience in the Summer and Winter Olympics and Paralympics and commemorative events such as state funerals and the War of 1812. In 2013, the program expanded to include thematic collections and events-based collections such as COVID-19, the 2022 trucking convoy protest in Ottawa, and current Canadian perspectives from the war in Ukraine. The program conducts rescue and preservation harvesting and ensures that federal government websites are preserved, pending decommission or content removal to another website. 

So who does all this work at LAC? The web archiving team consists of digital librarians and archivists that bring their own unique perspectives to web archival curation. On the technical side, the program’s senior crawl engineers are a critical component in tackling complex quality control issues. Tom and his team ensure that data curation aligns with user requirements and includes a digital humanities perspective with the purpose of building datasets for data historians, researchers and scholars. Tom used the following example to explain how the the digital librarians and digital archivists approach collection curation: “We ask the question, ‘Twenty years from now, when a digital historian sits down to write about the history of COVID-19 and its impact on Canada, what kind of data and sources do they wish they would have?’ This influences how we select for curation.” COVID-19 has demonstrated that web archiving is a key resource for  documenting history and the pandemic has influenced how they collect materials. 

LAC’s COVID-19 web archive consists of over 2000 resources, 16 TB of data and over 478 million objects, including 34 newspapers in both official languages representing various political and regional perspectives from coast to coast – and it continues to grow. The collection includes social media and over 4 million tweets concentrating on COVID-19 dialogue and its impact on Canadians.

Tom discussed the “black hole of quality control” (QC) and described the importance of using a methodological approach when conducting QC. He explained the importance of both a framework and a balance as QC can be a never ending project. For more on LAC’s approach to quality control, you can view one of the 2021 IIPC presentations from Tom Smyth and Patricia Klambauer The Black Hole of Quality Control: Toward a Framework for Managing QC Effort to Ensure Value

The importance of web archive finding aids was a key thread throughout the talk and later in the discussion. As Tom explained, finding aids enable researchers to see at a glance whether the collection or datasets are helpful in addressing their research questions. Ideal finding aids would include practical information such as a master seedlist, how many seeds for each theme, resource distribution, type of resource, language, QC status, and metadata specifications. 

The Prime Minister of Canada’s website ( was used as an example to describe why web archiving is so important. Back in 2006, LAC had only five days to ensure that all of outgoing Prime Minister Paul Martin’s website was captured before the site was replaced with the website of newly elected Prime Minister Stephen Harper. Tom underlined the importance of being proactive and the need to have good working relationships with partners and government departments such as the Privy Council Office, which supports the Office of the Prime Minister. 

The Truth and Reconciliation Commission Web Archive (TRC) is another example of collaboration between LAC and other institutions. LAC worked jointly with the University of Winnipeg Library, the University of Manitoba Libraries and the National Centre for Truth and Reconciliation (NCTR) to develop this collection. The relationship with Indigenous Peoples is a concentration for LAC and a constant subject of collection. All of these collections will be publicly findable in the Government of Canada Web Archive, which is planned to be released in the fall of 2022. 

Tom discussed the ethical and legal considerations of web archiving. The Library and Archives of Canada Act (S.C. 2004, c. 11, s.8(2)) empowers LAC to collect pertinent web content. Steps are taken to ensure the copyright owner is informed that their site will be archived and takedown requests are respected. Even with LAC’s extensive web archiving experience and resources, some websites just can’t be crawled due to their structure and the need for human input. 

LAC is a client of the Internet Archive and uses the Archive-IT software.  All of the WARCS are transferred back to LAC for local digital preservation via LTO tape. 

Tom closed his talk on a hopeful note. Our efforts today will help us to reduce the likelihood of what Vint Cerf has termed a “digital black hole” and “the forgotten century.” The global community of web archivists, through their concerted efforts, are making every effort to ensure that nationally relevant materials are captured, preserved and made available for future generations.

Library and Archives Canada Coffee Chat: Video and Slides

On Tuesday, May 10, 2022 the SAA Web Archiving Section was honored to host Library and Archives Canada’s Tom Smyth discussing web and social media preservation.

A video of the coffee chat and presenter slides will be available for a limited time.


Library and Archives Canada Coffee Chat

Presenter Slides

Web and Social Media Preservation Program at the National Library and Archives Canada

Coffee Chat Recap: U.S. Federal Government Web Archiving

By Susan Paterson, SAA Web Archiving Section Vice-Chair and liaison librarian at Koerner Library at the University of British Columbia Vancouver.

The Web Archiving Section hosted its first coffee chat of the year on March 29, 2022. Melissa Wertheimer, Chair of the Web Archiving Section, led a panel discussion on US Federal Government web archiving activities. We were joined by web archiving experts from the National Library of Medicine, Government Publishing Office, the Smithsonian and the Library of Congress. Topics ranged from content curation, collaboration amongst agencies, staffing, workflow models, and successes and challenges, as well as the impact of COVID-19 on their collecting activities.

The coffee chat recording is available to viewers until summer 2022.

National Library of Medicine – History of Medicine Division, Digital Manuscripts Program

The session began with an informative presentation by Christie Moffatt, Manager of the Digital Manuscripts Program, History of Medicine Division at the National Library of Medicine (NLM). Christie described NLM’s approach to web collecting, an activity which began in 2009 with their Health and Medical Blog Collection. Christie provided an overview of the NLM web archive collections, and the collection development policies that guide their web collecting.

Christie noted that thematic and events based collections have grown and are a key component to their collection building. The Global Health Events Web Archive collection, one of their largest, began in 2015 with the Ebola outbreak. With the World Health Organization’s announcement of a global health pandemic in January 2020, NLM started work on a COVID-19 pandemic collection. Interestingly, this specific designation of a global health emergency is worked into their collection development policy and the NLM takes responsibility for building a web archive once this designation is made. An aim of the COVID-19 collection is to ensure a diversity of perspectives on both the impact and the response to the pandemic are archived. Tools and communication used for outreach during the pandemic (i.e. TikTok, Twitter) as well as personal narratives of the pandemic make up part of the collection which is the Library’s largest collection with 3.5 FTES working on the project. The National Library of Medicine’s Archive It website can be explored here.

Government Publishing Office

Dory Bower, Archives Specialist at the Government Publishing Office (GPO) provided an overview of GPO’s web archiving activities and explained how legislation Title 44 of the U.S. Code is the mandate for Public Printing and Documents. Specifically, Chapter 19 discusses the Depository Library Program. Federal agencies should be distributing and publishing their documents through the GPO. If a federal agency is not doing so, the Superintendent of Documents should be notified. Unfortunately, with born digital publications and publishing directly onto websites this is not happening and material is being missed. GPO joined the Archive-it community in 2012 and since then they have been using web archiving to help fill this gap.

The web archive is part of the overall GPO Digital Collection Development Plan. For material to be part of the GPO collection, it must be federally funded and be within the scope of the Federal Depository Library Program (FDLP). GPO is also focusing on collections of interest geared towards women, tribal libraries, and Native American communities, just to name a few. GPO maintains over 213 collections in Archive-it, making up 38.3 TB of data and consisting of over 392 million urls. You can explore the Federal Depository Library Program Web Archive here.

Smithsonian Institution – Libraries and Archives

Lynda Schmitz Fuhrig, Digital Archivist at the Smithsonian Institution Libraries and Archives, presented next. Lynda provided a fascinating overview of the web archiving work that’s being done at the Smithsonian, which celebrated its 175th anniversary in 2021.

The Libraries and Archives of the Smithsonian is the official repository and record keeper for the Smithsonian. Their responsibilities include sharing, collecting, and preserving Smithsonian activities and output which includes documenting its unique web presence, which launched in 1995. They now have nearly 400 public websites and blogs and a very active social media presence covering Twitter, Instagram, Flickr, and YouTube – just to name a few.

Like the National Library of Medicine, the Smithsonian has documented the impacts and effects of the COVID-19 pandemic on America. For example, beginning in March 2020, more focused and frequent crawls were necessary to document the altered scheduling of the closing and reopening of the Museums and the Zoo. Additionally, the closure of museums created a need for an increased digital presence, and the Smithsonian launched several new websites and initiatives, including Care Package, Vaccines & US and Our Shared Future.

Audience members were particularly interested in their web and social media workflows and tools. Along with Conifer and Browsertrix, the Smithsonian uses netyltic, which was developed by the Social Media Lab at Ryerson University to collect Smithsonian hashtags and accounts.

They are one of the few organizations that download the WARCS from Archive-it. They developed an in-house tool called WARC Retriever which they hope to release on Github later this year. Lynda’s summation was poignant: “The Smithsonian web collections will continue to tell the history and stories of the Smithsonian.” You can explore the Smithsonian Archive-it page here.

Library of Congress – Web Archiving Program

To round out the panel, Meghan Lyon and Lauren Baker, Digital Collection Specialists from the Library of Congress (LOC) Web Archiving Program, provided an overview of the activities at LOC. The Web Archiving Program began in 2000 and is part of the Library’s Digital Content Management Section.

The LOC web archives consist of 3PB of data organized into 174 collections, 75 of which are active collections. Like many of the other speakers, the Web Archiving Team collaborates frequently on web archive collections, relying on the contributions of collaborators around the Library. The Collection Development Office helps guide collection initiatives, and various committees review subject proposals and select content to archive and determine collection focus. LOC comprehensively collects content from Legislative Branch Agencies and U.S. House and Senate offices and committees. They collect content about U.S. national elections as well as managing other events based and thematic collections.

Megan and Lauren addressed the issue of permission and web archiving. Their permissions policy is determined by the Office of the General Counsel, which is based on Copyright Law. Permission requests must be sent to site owners for anything selected for web archiving. There are two permission requests: a permission to crawl, and permission to display based on the country of publication and the category of the entity. You can explore the LOC Web Archiving Program website here.

The panel closed out the session by discussing how they became interested in web archiving and how their careers started in the field. Their initial experiences ranged from practicums to learning on the job. The remainder of the conversation also included the topics of trends, the and future of web archiving tools – including what improvements people hope for and imagining better tools for harvesting and user awareness. The session was well attended with 181 registrants and over 80 attendees. Thank you to everyone who presented and who attended for such an engaging hour. Stay tuned for our next coffee chat, which will be in May!