“We gathered the data.”
“The documents were collected online.”
“We retrieved records from the archive.”
“Relevant articles were identified through keyword searches.”
“The material was obtained from publicly available sources.”
“We downloaded the newspaper articles.”
“A dataset was compiled from institutional websites.”
“We accessed the collection through the library’s online portal.”
“The corpus was assembled from digitized documents.”
“The resulting data were cleaned and analysed.”
Seeing any of these sentences in a paper discussing hundreds, thousands, or even millions of documents leaves me wondering how the material was gathered and curated. In my experience, the institutional and library databases I use usually have no API (a way for a program to request records from the database directly) or bulk-download tool to make that work easier. The Korean History Database (db.history.go.kr), run by the National Institute of Korean History, is a good example. It holds an impressive amount of material, but it supports neither API access nor bulk downloading. I doubt anyone would put themselves or their student assistants through the task of downloading each document one by one.
More likely, many of these studies used a program that did this work for them, faster than humans could. A program that visits web pages, follows links, and saves the documents or extracts the information you ask it to collect can make this repetitive process a lot easier. This is called a scraper. A scraper downloads content, often a large amount of data. This can be anything from X posts, images or comments. In my case it was digitized primary sources dating to colonial Korea.
Academics have done this for a long time. In 2013 the Programming Historian published a lesson by Kellen Kurschinski that downloads 1,487 page images of the Thomas Jefferson Papers from the Library of Congress with a single wget command.1 Five years later, after Facebook closed the API interface researchers had used to collect posts from public Facebook pages. Following this, Deen Freelon urged researchers who study social media to “learn how to scrape the Web,”. Freelon also pointed out how troublesome this was and sometimes still is as it “differs for every website.”2
The fact that the scraping process was different per site and also prone to the website changing is what coding agents now take on. You describe in plain language what you want collected, including a sample URL. The agent reads the pages, writes the scraper, runs it and fixes it when it breaks. Simon Willison describes for example how he asked Codex, from his phone in a coffee shop, to collect a conference schedule. Thirteen minutes later it returned the script and the data.3 An academic without programming training can now build a scraper the same way. Checks are important. Agents can potentially make mistakes. Increasingly, however, agents are able to check this type of work quite well, especially since the failure and success conditions of scraping (gathering all the data you asked for) can be checked against the page.
While it is easier than ever to scrape content of your interest, openly sharing in an academic setting that you did still remains a stigma. In a March 2024 editorial, Alejandra Alvarado and Andrew Hufton, the editors of the Cell Press journal Patterns, wrote:
At Patterns, we frequently encounter authors who have used web-scraped datasets but are unclear on how to meet the journal’s transparency and reproducibility requirements. These cases can raise complex questions regarding data origin, copyright, and ethics (see Paullada et al., 2021, Patterns) for what may otherwise appear to be straightforward research articles.4
I suspect some of the vagueness in provenance statements like the ones at the top of this post comes from researchers being unsure where they stand. A website often allows you to read and download its documents “manually” while discouraging automated collection. If you used a scraper, would explaining that in a paper invite questions about whether you were allowed to collect the material in the first place? Could drawing attention to the scraper lead the institution to restrict access, making the material harder to obtain for other researchers too? I can see why someone might hesitate if they worry that being explicit could cost them access, expose them to a legal complaint, or leave them unable to use or share the dataset their research depends on.
The Wild West
An excellent piece by Eun Seo Jo and Timnit Gebru maps a scale of supervision in data collection, mainly for the machine learning field.5 Their figure places “’Wild West’ Web Crawling” at the laissez-faire end of that scale and curatorial archives at the interventionist end. Jo and Gebru report the nickname from other researchers. They write that the lack of any systematic process for building datasets “has spurred researchers to call it the ‘wild west’.”6 Their paper asks machine learning to learn from archivists. This post asks people who work with archives to borrow the scraper’s tools. Jo and Gebru go into great depth on many issues that I can only briefly mention here. I recommend you read their paper.
Their example of the laissez-faire end is face recognition trained on photos taken from Flickr.7 Those of us in the humanities and social sciences are not per se interested in facial analysis. Rather, we are sometimes stunned by the sheer volume of material now being made available through digitization efforts or otherwise. It is this material that is less sensitive in a privacy, compliance and copyright standpoint that this post focuses on. Most problems lie with data such as social media posts and training sets built from living people’s photos and words are where the cases get more difficult. I leave these out for now as I want to draw attention on the cases where scraping and dataset curation is less of an issue.
While I encourage you to consider scraping sources of your own interest, scraping digitized archives still has costs. A scraper firing rapid, repeated requests does the same work as a denial-of-service attack and can overwhelm a website for everyone else.8 That is true for a national library as much as for a small scale archives. Their servers are shared with every other user. Running a scraper for days that sends them a great many requests can impact their services. It is important to be “polite” by building in rest times, or other methods that cap the speed. In other words, “be a responsible internet citizen.”9
While for now this is true, it is also important for institutions to think of ways to facilitate access to researchers and those of interest to get access to the data without resorting to the scraping. This process usually takes a few days, and researchers don’t seem to mind waiting for their data to become available to them. Gating access, limiting API speeds etc to times where usage is low for example seems like a good load balancing approach while still allowing access.
Copyright law is still a factor to consider. A database can carry right(s) of its own on top of the copyright in the documents it holds. Alvarado and Hufton warn that the right to mine copyrighted material “does not necessarily grant you the right to store and share that material.” For example, in the EU, Article 3 of Directive 2019/790 lets research organisations with lawful access copy and extract works for text and data mining, covering database rights as well as copyright. It allows them to keep those copies “for the purposes of scientific research, including for the verification of research results,”. It says nothing about publishing them again. The same article lets the archive take measures to protect “the security and integrity of the networks and databases,” so a scraper that strains the server can expect to be throttled or blocked.10
Riding out anyway
What I’m about to advocate for in this post is that you ride out with me, and with the many who are already out West, in search of ever more data riches on the web. I do this without forgetting the responsibilities we have, which the editors of Patterns ask us to keep in mind. Nor do I want to deny any of the gray areas or issues described by Jo and Gebru and many others. I do think, however, that this “Wild West” is also a land of opportunity. Especially since everyone can now gain access to it through scraping.
By owning your data I do not mean owning the documents. Downloading a newspaper page does not give you its copyright, nor the right to publish it again. What I mean is keeping your own copy of the sources your research depends on. A copy on your own machine can also be searched in ways a portal cannot. A portal’s search box usually takes one query at a time and can itself be influenced by the manner in which its search function operates. An agent pointed at a local copy can run many searches from different angles, in hangul, hanja and romanization, related terms etc, with the spelling and name variants a colonial-era document might use. It can also keep a record of every query it ran.
If you take care of your copy in this way, rewards follow. You keep access to your sources. You can search them faster and more widely. Others can check your work.
Archives can go dark
There is also another important reason to keep your own copies. In October 2023 a cyberattack destroyed large parts of the British Library’s technical infrastructure. The collections themselves survived, but the Library is still recovering the systems around them. EThOS, its Electronic Theses Online Service, went offline after the attack and stayed unavailable for more than two and a half years. On 8 July 2026 the Library restored the metadata, now over 650,000 thesis records. For the full text, readers now depend on university repositories.11
Data I and you might rely on can be taken offline, replaced or removed. I don’t trust that the data I need will always be available to me. Nor should you. Scraping is first of all a way to make sure the data no longer lives in only one place, on one institution’s server.
Yes, the chances of this happening to your or my data are slim, but not zero. More likely causes might be budget changes that force institutions to make choices they’d rather not make. But I’m no longer waiting to find out. To me, scraping is a way to avoid losing access, temporarily or permanently, to data that is important (or even vital) to my research.
See you out West
So here we are. A lot of sources are online, scraping is easier than ever, and getting your own copy allows you to go beyond what you might have traditionally attempted.
Through scraping and gaining access to a large body of sources my research and workflow are better, safer and faster. The tools are there now. An afternoon with a coding agent is enough to turn a collection into a copy on your own machine.
There is plenty of room out West. I’ll see you there.
Kellen Kurschinski, "Applied Archival Downloading with Wget," Programming Historian (2013), https://doi.org/10.46430/phen0022. The command waits two seconds between files (-w 2) and limits the download rate (--limit-rate=100k).
Deen Freelon, "Computational Research in the Post-API Age," Political Communication 35, no. 4 (2018), 665-666, https://doi.org/10.1080/10584609.2018.1477506. The closure he describes is that of Facebook's Pages API on 4 April 2018.
Simon Willison, "Vibe scraping and vibe coding a schedule app for Open Sauce 2025 entirely on my phone," 17 July 2025, https://simonwillison.net/2025/Jul/17/vibe-scraping/.
Alejandra Alvarado and Andrew L. Hufton, "From Scraped to Published," Patterns 5 (8 March 2024), https://doi.org/10.1016/j.patter.2024.100953.
Eun Seo Jo and Timnit Gebru, "Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning," Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20), https://arxiv.org/abs/1912.10389. The scale is their Figure 2, "Scale of Supervision in Data Collection" (arXiv version, p. 3).
Jo and Gebru, "Lessons from Archives," arXiv version, p. 2.
Jo and Gebru, "Lessons from Archives," arXiv version, pp. 3-4. They cite an NBC News report on facial recognition datasets built from millions of online photos "scraped without consent" (Olivia Solon, 2019). Alvarado and Hufton likewise note that facial image datasets "are under scrutiny for their potential abuse and lack of proper consent processes."
Jonathan Paige, "The Legality and Ethics of Web Scraping in Archaeology," Advances in Archaeological Practice (May 2024), 100, https://doi.org/10.1017/aap.2023.42.
Paige, "Legality and Ethics of Web Scraping," 100. He names the R package polite as one such tool.
Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market, Article 3(1)-(3). Article 7(1) makes contract terms that override this exception unenforceable. Whether a foreign website's terms of use count as such a contract is a question for a lawyer.
British Library, "EThOS," https://www.bl.uk/collection/ethos (consulted 15 November 2026).


