Earlier this month, a thread on X that went quite viral pointed out that a prominent political scientist had a half-dozen fabricated citations in a recent article on South Korean politics.1
The reaction was about what you would expect: disappointment, mostly, and in some quarters shock. No instructor would accept this from a student, and when it appears in a published article it has to be addressed, through a correction or a retraction. Given the number of bad references, I imagine most people following the case expect the latter.
Still, the speed and the intensity of the response are worth pausing over. A case like this should have been handled with more care.
First of all, we do not know how the citations got there. It is unlikely, but it is possible that the fault lies with the journal rather than the author. Many of us have concluded that this outlet does not meet the standards it advertises, and at the time it was turning articles around very quickly to get thoughts out on the fast-moving events in Korea. It would not be out of the question that an editor ran the article through an AI program to tidy or reformat the references, and that the program made them up. I do not know whether this is what happened, and it seems like this information should have come out by now. But neither does anyone else.
It is fair to call out errors of this kind. But it is also important to be certain that the grounds for a public accusation of malpractice against an individual are solid, because once it is made, it cannot easily be undone. The damage to reputation is immediate and severe, and a later correction, if one comes at all, rarely gets the same attention.
It is also a time for humility. In the most mundane and boring parts of academic work, such as locating a source and formatting a reference, many of us have had the rug pulled out from under us. These tools are very good at making it seem like the work is done correctly when it is not, and they can trick even otherwise careful people into thinking that their personal robot got it right.
I suspect that much of this anger is coming from a broader anxiety about what AI is doing to the academy. All that to say, we should be far more sure of what happened before we make an example of someone who has, by and large, done excellent work, and we should consider the case in the context of larger structural issues.
To be clear, I do not actually know how this happened. But I am far more inclined to believe that it was a case of misplaced confidence in a technology that routinely spits out convincing falsehoods than a case of calculated fraud. That does not excuse the error, but it does change what we should learn from it. The failure may be less an individual lapse than a structural one: a culture that has adopted powerful tools without equally powerful norms of verification. The good news is that this is a problem we can fix — and I have some solutions to try out. But first, we need to understand how we got here, and why the machine is so good at deceiving us.
Old habits die hard
Type a scholar’s name and a couple of keywords into Google Scholar, and you get a list of citations. Use a few of them without reading past the abstract because you already know roughly what that person argues, and you have done one of the more mundane tasks of writing. AI is just exposing an inconvenient truth. We do not (closely) read every source we cite. Some citations are indeed used as evidence, and some are presented as a kind of performance. They signal that we know the literature or that we have cited whoever a reviewer expected or explicitly asked us to. A chatbot like ChatGPT or Claude, in these early days, feels like the same exercise. You ask for the literature on a subject or particular claim, and it gives you names you recognize along with titles that look right. The problem is that some of those titles do not exist.
None of this is new. Take one prominent example. A decade ago, before anyone could blame a model, a prize-winning history of North Korea was found to cite nonexistent or irrelevant sources in at least sixty-one places. The author returned the prize and left his chair.2 A milder form of the same problem is extremely common: a wrong DOI, a page range off by two, a citation to a real paper for a claim it does not make. These are hallucinations in all but name. What AI adds is ease of use and an appearance of accuracy, and that is one of the big structural issues we are dealing with now.
Why the machine makes things up
Google Scholar (usually) returns sources that actually exist. A large language model returns what is probable. It is not a database, nor a search engine. It spits out the next most likely token given everything before it, so when you ask for a citation, it produces a string that looks like one: an author, a year, a title, a journal, and a volume, because that shape is overwhelmingly probable wherever citations appear. Nothing in that process checks the reference against a master list. The model does not care whether the work exists, which makes it a bullshit machine in Harry Frankfurt’s sense of the term. This problem is deeply entrenched in these models, as the authors of a recent paper from OpenAI point out. When we train and evaluate these models, we usually end up rewarding a confident guess over an admission that the model does not know the answer.3
How often does this happen? In one controlled study, GPT-3.5 fabricated 55 percent of the citations it generated for literature reviews. The newer GPT-4 was much better, but still fabricated 18 percent.4 In an audit of two and a half million biomedical papers, the rate of fabricated references rose more than twelvefold after 2023, to about 57 per 10,000 papers by early 2026. The steepest rise came with the introduction of new AI writing tools in mid-2024. Most of these references had made it through peer review, which is understandable, since it would be a lot of work for reviewers to check every source.5 The problem reaches well beyond the academy. In 2023, a federal judge fined two New York lawyers five thousand dollars for submitting a brief that included six judicial opinions ChatGPT had invented. One of the lawyers had used ChatGPT to verify that one of the opinions was real.
We do not know how the fabricated citations in the case described at the start of this post were produced, but my guess is the simplest one: someone used a single prompt to a general-purpose chatbot, with no tools or documents to check against. That is the weakest possible way to use the new technology, and it almost guarantees some hallucination in the answer(s). Tools that retrieve and read sources before answering do much better, but they still have a way to go. When Stanford tested retrieval-based legal tools that vendors marketed as “hallucination-free,” they still gave a false answer in something like one out of six cases. But this is at least a sign of things moving in the right direction.
At the very least, we need verification systems that can find hallucinations and other errors.
Responsible all the same
We are accountable for what we publish. The best way to avoid this problem is to read the sources we use carefully, check every citation, and leave out anything we cannot verify. Fake sources are unacceptable whether they come from misusing an LLM, a tired research assistant, or our own carelessness in formatting.
For students, that standard has to be explicit, and at my institution it is. I am not impressed by how universities (mine included) mostly cast AI as a tool for cheating, but Leiden University’s Faculty of Humanities gets the definition of fraud right: any act or omission that prevents a correct assessment of a student’s knowledge. Its examples include the unauthorized use of AI software and the invention of research data.6 Its guidance on generative AI is more direct: “GenAI generates language, not information … If asked to produce references, many LLMs will simply invent non-existent citations.” A fabricated citation misrepresents the evidence the work rests on, which is exactly what assessment is meant to evaluate. So yes, if taken literally, it is fraud, and the heaviest penalty on the books, exclusion from all examinations for a year, should be deterrent enough.
A field-wide response is going to be harder than a local one, and the distance between what these policies aim to resolve and what actually happens seems quite wide. It seems like people are trying to police their way out of this problem. That comes from a good place, a wish to make sure people can do good science, and it is understandable. In mid-May, for example, arXiv announced that authors who post papers with hallucinated references will be banned for a year, and after that can only post papers that have cleared peer review elsewhere.7 I like the motivation, but not the mechanism. The policy is, as has been noted, “welcome but unenforceable.”
The early evidence backs this up. When one peer-review service ran its reference checker over two hundred papers posted to arXiv after the ban, more than a quarter of them still included a hallucinated citation. Despite the threats of punishment, it is obvious that people are using AI as a research tool. The technology is too good and too useful for prohibition to hold. People are going to use it to think with and write with, whatever the rules say. To the extent that the deterrent messaging is heard at all, I think it is mostly leading to underreporting of AI use because of social desirability.
We need a different approach. The answer is to teach people to use the tools openly rather than in secret, and to turn the same tools on the problems they have made worse. We can, for example, build them into the checking process when someone submits work to a journal. We can also use them to help writers, senior authors and students alike, audit their reference lists and check that each source actually supports the claim attached to it, until this becomes routine. Done properly, this checking could be more systematic and reliable than it ever was before ChatGPT. Rather than tell people what they cannot do, we should show them what they can do instead.
Skills / chatbots
The first thing we need to do is stop using chatbots as the default for serious research and writing, and start building (or using) skills for agentic AI instead.
A skill is a set of instructions that an agentic AI system uses to perform a task. A skill constrains what the AI can do by giving it specific steps to follow and the tools to use for those steps. It also gives the model steps to check its own work. With a one-shot prompt, we are asking the AI to be creative. With a skill, we are taking that opportunity away. A skill can tell the model to dispatch agents, to look up sources in actual indexes, and to tell the difference between what it confirmed and what it merely guessed. Fabrication comes from a model answering from probability instead of evidence, and a well-built skill helps close off that option.
I write these for my own research and keep them in a small library, the Open Science Skills. Three of them are relevant here.
The first is a reference checker (citation-check). This one most directly addresses the problem discussed in this post. It reads every in-text citation and every bibliography entry in a paper and then checks each reference. It confirms that the reference exists by looking it up in programmatic indexes such as Crossref and OpenAlex, not in the model’s memory. For references with a DOI, it checks that the DOI resolves to the cited paper and not merely to some paper. If a title looks made up, it checks it against the named author’s actual body of work. Only if it cannot find the reference by title or by author does it call it fabricated. What comes back separates the serious problems from the cosmetic ones and flags the uncertain cases for a human. It is not perfect, but overall, it works.
The second skill addresses a different, arguably much deeper problem, one that is much less likely to go viral on X. Even if a citation is real and formatted correctly, what if it does not actually support the claim it is attached to? The source-claim checker (fact-check) reads the cited source and asks whether it supports the specific sentence, and it flags overclaiming and direction errors. To do that, it needs the sources to read. As I’ve written the skill, it cannot just look things up on the internet or “get creative.” That is what the third skill is for.
The third, a knowledge-base scaffold (research-repo), builds a project around its source library. The library has two folders: one for the original source documents (PDFs, etc.) and one for the machine-readable versions (preferably in markdown) that the checker actually reads. Each document is keyed to an entry in the bibliography, so that each source is directly connected to its bibliographic metadata. The claim checker will not run against an empty or half-built library, because to verify that you have represented a source faithfully, you need to have that source, or, where the source is not available, a file that faithfully summarizes it.
Try it yourself
In practice, none of this is as elaborate as it might sound, and the best way to figure it out is to do it yourself. For this post I put up a small, self-contained walk-through on a synthetic manuscript seeded with deliberate errors, so you can watch the checks fire. It is the first of a series of demos I am preparing for students and colleagues at Leiden University in a teaching hub (still a working prototype), with the setup guide and slides alongside it.
We are in an odd spot right now. The technology is too useful to ignore, but still unreliable enough to embarrass or even discredit us. And we have not been at it long enough to establish good habits. That is a dangerous combination.
What we need is not a moratorium, or a rejection of the technology, or a hunt on social media for the next person to make an example of. What we need is humility about how easily these tools can fool us, some grace for those who have misused them, and a serious tightening of the standards for how we use them. These positions are not inconsistent. We can hold all three at once.
Correction: It has since been clarified that the article published in Korea Observer was an "annual review" reflection and was therefore not subject to peer review, which explains its quick publication timeline.
The article appeared in Korea Observer 57, no. 1 (Spring 2026): 1–21, https://doi.org/10.29152/KOIKS.2026.57.1.1, a 21-page review of South Korean politics after the December 2024 martial law crisis. It was submitted on February 10 and accepted on February 21, and the journal advertises itself as a peer-reviewed quarterly. The widely shared thread on June 13 flagged six citations as fabricated, and two of the scholars named confirmed the attributed works do not exist. Two of the named authors, asked by news outlet NK News, confirmed they had never written the works attributed to them. The article published, notably, was not subjected to the journal’s peer review process, as it is a special “annual review” reflection.
Charles K. Armstrong, Tyranny of the Weak: North Korea and the World, 1950–1992 (Ithaca, NY: Cornell University Press, 2013). The citation problems were first documented in detail by the historian Balázs Szalontai. A Columbia University investigation found research misconduct, concluding that Armstrong had “cited nonexistent or irrelevant sources in at least 61 instances,” alongside plagiarism; Szalontai’s own count ran higher still. In 2017 he returned the American Historical Association’s John K. Fairbank Prize, writing, “Due to the numerous citation errors in my book, I have decided to return the prize out of respect for the AHA,” and he retired from Columbia in 2020.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang, “Why Language Models Hallucinate,” arXiv:2509.04664 (2025), p. 1: “language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty … language models are optimized to be good test-takers, and guessing when uncertain improves test performance.”
William H. Walters and Esther Isabelle Wilder, “Fabrication and Errors in the Bibliographic Citations Generated by ChatGPT,” Scientific Reports 13 (2023): 14045, https://doi.org/10.1038/s41598-023-41032-5, p. 5. Across 636 citations, “55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated. Likewise, 43% of the real GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors.”
Maxim Topaz, Nir Roguin, Pallavi Gupta, Zhihong Zhang, and Laura-Maria Peltonen, “Fabricated Citations: An Audit across 2·5 Million Biomedical Papers,” The Lancet 407, no. 10541 (2026): 1779–1781, https://doi.org/10.1016/S0140-6736(26)00603-3. The audit scanned the PubMed Central open-access subset, about 2.5 million papers and 125.6 million references, of which 97.1 million carried an identifier and could be verified. It found the rate of fabricated references rising more than twelvefold, from roughly four per 10,000 papers in 2023 to 51.3 per 10,000 in the fourth quarter of 2025 and 56.9 in early 2026, with the sharp inflection in mid-2024. The share of papers with at least one fabricated reference rose from one in 2,828 in 2023 to one in 277 in the first seven weeks of 2026. The authors cite earlier work estimating that 30 to 69 percent of LLM-generated references in biomedicine are fabricated.
Leiden University, Faculty of Humanities, Course and Examination Regulations, Art. 7: “Fraud is defined as any activity or omission carried out by a student aimed at completely or partially hindering a correct assessment of the student’s knowledge, understanding and skills, including at least the following: … c. unauthorized use of AI software; d. inventing research data …” The enumerated list does not use the words “fabricated references” verbatim, so mapping a hallucinated citation to a specific clause is an interpretation, but coverage under the general definition is clear.
Reported across Nature, TechCrunch, Times Higher Education, Inside Higher Ed, and others in mid-May 2026, and announced by an arXiv moderator and computer-science section chair rather than, at first, as codified policy text on arXiv’s official pages. The stated trigger is “incontrovertible evidence” of unchecked LLM output, with hallucinated references the main example. This is distinct from arXiv’s earlier (October 2025) requirement that review and survey articles in its CS category clear peer review before posting, which is a submission rule, not an author ban.





Welcome to Substack, Steve. I write a bit about AI over here, too: https://www.collegetowns.org/p/ai-wrangler-job-of-the-future-combines