CILIPS Professional Development Fund Report – Web Archive Research in an age of Smart Data at the British Library
Category: Blog
Written by Eilidh MacGlone, Web Archivist, National Library of Scotland.
I archive websites under legal deposit for the National Library of Scotland (1). NLS preserve Scottish websites, and UK websites of interest to Scottish readers under the Legal Deposit Libraries (Non-Print Works) Regulations 2013. Websites contain Scottish culture and we have
a responsibility in law to collect and look after these publications and to support research..
In the hope of gaining new insights into what researchers want, I attended an event held at the BL – generously funded by the CILIPS Professional Development Fund.
Our chair was Professor Jane Winters, who spoke about web archives as “an extraordinary primary source, especially when it can be work with as data and smart data.”
The first speaker, Thom Vaughn, introduced us to Smart data, giving us examples such as extracted text datasets, web graphs showing connectivity, language identification datasets, domain-level and host-level analyses and metadata and indexes, which enable targeted research. His talk went on to describe how Common Crawl are an often seen, polite bot on the web, gathering mainly material in English, though aiming to be more inclusive of other languages.
Emmanouil Tranos presented the work of the Building an Atlas of Economic Activities project which attempts to create a framework to understand digital infrastructure, as actual infrastructure. It utilises the JISC UK Web Domain Dataset which indexes all archived web pages from the .uk domain. He described a study of Shoreditch in London, identifying businesses over time. Their finding was that this research surpassed analysis based on official data, which does not capture some modern business activities. The ONS has recently run a consultation on an update of SIC (2). The project plans to produce their dataset as Findable, Accessible, Interoperable, and Reuseable (FAIR) data product (3).
Andrea Kocsis, previously Librarian’s scholar at NLS, spoke about the Digital Ghosts exhibition, which began with data from Trevor Thomson and me, combined with IA data to create a “Scotland on the Internet dataset.” Andrea Kocsis explained her process of developing Digital Ghosts, moving from and eventually abandoning data science to working with artist Dorsey Kauffman and software designer Parker Kauffman to exhibit Trevor Thomson’s data as an art object, in a way which would be meaningful for the public.
Cameron Hugget, a Collaborative Doctoral Student at Teesside University and the British Library, spoke about insights gained from close reading of comments on webpages. His work examined racism in physical copies of zines from the seventies to the 2000’s. In particular, posts on a fan forum made in response to a dispute between Anton Ferdinand and John Terry and news reporting around it, corroborating these against other threads and print fanzines. He set out his technical insights on his work clearly, that he required reproducibility, access and preservation of web, saying: “This type of culture really matters … the internet is real life.”
I felt privileged to hear about this work to make sense of datasets and writings, which are not always easy to develop methods for – a lot of my admiration comes from how creatively the researchers approached this problem. I felt confirmed in my goal of providing readers with datasets which they can read, reproduce, take home and publish, and be read by others.
1 “Legal Deposit Explained.” National Library of Scotland, https://www.nls.uk/about-us/who-and-what/legal-deposit/legal-deposit-explained/. Accessed 17 Feb. 2026.
2. SIC and SOC – How the ONS Is Updating Its Industrial and Occupational Frameworks | National Statistical. https://blog.ons.gov.uk/2026/02/16/sic-and-soc-how-the-ons-is-updating-its-industrial-and-occupational-frameworks/. Accessed 17 Feb. 2026.
3. Wilkinson, Mark D., et al. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data, vol. 3, no. 1, Mar. 2016, p. 160018. www.nature.com, https://doi.org/10.1038/sdata.2016.18.