Online panels have long had a quality problem. Some respondents speed through questions (speeding) and some give the same answer to every question (straightlining). Attention checks are the standard way to catch them, but what about survey takers who automate answering? In November last year, Sean Westwood published a paper in PNAS showing that his AI agent could answer online surveys in a consistent persona and pass 99.8 percent of 6,000 common attention checks.1 If that holds, and automated survey completion becomes (or perhaps already is) common, the main tool for catching low-quality respondents no longer works, and any study run on an online sample is compromised.

In August, I fielded survey experiments on attitudes towards immigration in four countries (Germany, South Korea, Singapore and the US), with around 2,500 completes per country from Bilendi (Germany), Rakuten Insight (South Korea and Singapore) and Prolific (US). The core of the survey was two conjoint experiments, in which respondents evaluate a series of profiles whose features are randomly varied. In the first, respondents saw one immigrant at a time and said whether they supported granting that person permanent residence. In the second, they saw an immigration policy that varied in how it was decided and implemented, the area it covered, its direction and the group it affected, and said whether they supported it. Each experiment had eleven tasks, and the last repeated the first exactly. With Westwood’s result in mind, I did not rely on attention checks alone. I added layers of checks for both AI use and low effort. This post looks at the wider problem and then at what I found, or didn’t find, in my own surveys.

Subscribe now

What do we know about AI use in surveys?

How large this problem is likely to be is a matter of some debate. Westwood and Frederick flagged 4.4 percent of a sample from Prolific for indicators of using AI (either pasting in text or typing too fast).2 Rothschild et al.’s response to their paper argued that this number was too high.3 They also point out that there are at least three distinct problems. Human respondents may use chatbots to help them answer, simple agents may answer on their own, and autonomous agents may answer from inside a managed panel. The first of these is well documented, and seems quite common. Veselovsky et al. find between a third and nearly half of MTurk workers using large language models (LLMs) to aid them in a summarization task,4 and Zhang, Xu, and Alvero find about a third of their respondents admit to using an LLM to answer open-ended questions.5 The evidence for the other two is thin, with Morning Consult finding signs of OpenAI’s Operator agent in around 0.01 percent of interviews.

There is very little evidence for the last two, but it is possible to check for all three (plus low-effort respondents), so I did.

What I checked

I have grouped the checks by what each was meant to catch. In most cases, the automation checks were run by a short script on four of the conjoint screens (the first task in each experiment, and again for the repeat).

1. Low effort: Attention checks have long been used to weed out respondents who are not paying attention.6 They are not much help against AI, though, as Operator passed Morning Consult’s attention checks.7 The repeated task gives a baseline of how consistent respondents are with themselves (about 77 percent on average across studies, according to Clayton et al.),8 and I recorded how long each screen took.

2. Chatbot help: Pasting is a common sign of chatbot use, and disabling copy-paste roughly halved LLM use by crowd workers in one study.9 The script counted pastes. Near the end of the survey I also asked directly, “Did you use an AI tool (such as ChatGPT, an AI browser assistant, or similar) to help you complete this survey?” This is obviously imperfect. An agent built to pass as human would presumably not admit it, while someone using a translation tool might. Still, Xu and Malkin found that most respondents who admitted to using AI had already been caught by a copy-paste check, and that agents usually admitted to being used when asked.10 People do, however, under-report their own AI use.11

3. Scripts and agents: The script recorded whether anything filled an off-screen “honeypot” text field, which a human cannot see but a script auto-filling every input would fill. Honeypots are inexpensive to add, but Storozuk et al. found none of their bots triggered them,12 and Operator was not fooled by them either. The script also recorded whether the browser reported being under automated control (the webdriver flag, which many automation tools set by default), and it counted mouse movements, clicks, key presses and touches. I could not find a study on whether consumer agents set the flag. Mouse movements say little about respondents on phones. Prolific has found that behavioral metrics of this kind correctly identify agents 95 percent of the time, compared to 60 percent for more classic attention checks.

4. Platform screening: Qualtrics scored each session with Google’s reCAPTCHA and a fraud and duplicate service, and Prolific runs its own checks to keep its pool authentic.

The automation checks came back empty

Among 10,930 delivered completes, zero participants filled out the hidden field, zero participants used a browser that reported being controlled by an automated program, and zero participants got through all of my measured screens without interacting with any part of the screen. Two participants (both in the US) pasted text. Neither reported using AI. reCAPTCHA flagged 2 to 13 percent of sessions as suspect, but because the reCAPTCHA score is not very specific, and the more direct measurements are all zero, I believe this is just noise. Prolific’s quality checks were similarly unremarkable. Among the 2,462 participants in my main US study, no participant scored Low (problematic) on the LLM check or the bot check, and 19 scored Mixed on the LLM check and 12 on the bot check.

Who admitted to using AI

In total, 854 respondents said they had used AI to complete the survey. This corresponds to 0.4 percent in the US, 4.6 percent in Germany, 8.6 percent in South Korea, and 16.5 percent in Singapore. If these were simple scripts, at least some of them would have set off the automation checks — but none did.

I did some digging to see if I could spot differences between the two groups. People who say they used AI take between a third and a half as much time to complete each conjoint screen in every country (a median of 8 to 10 seconds per screen against 21 to 29). They are also more likely to be using a touchscreen device, fail the attention check, and (in Germany and South Korea) not give the same answer to the same task a second time.

So the “typical” self-reporting person seems to be someone speeding through the survey and using a touchscreen device. If they were using a script I think it would show up in the paradata somehow, but I do not see any evidence for it. There is one small exception. The 11 US respondents who said yes barely moved the mouse (a median of 6 movements per screen against 256), though they also finished each screen in 8 seconds and none of them pasted or set off the automation flag. I do not have any data to confirm this, but it is possible that something about being on a phone makes people more likely to say “yes” (maybe some AI feature on the keyboard or browser?). Translation tools are another possibility, since the rate was highest in Singapore, where some respondents may have run the questions through a translator. I gave Singaporean respondents the option to take the survey in English or Mandarin Chinese (most opted for English), so I’m not sure about the translator explanation.

The self-report also overlaps with the checks I already had. In Germany, South Korea and Singapore, between 28 and 36 percent of those who said yes failed the attention check, against 3 to 8 percent of those who said no. Between 36 and 40 percent showed two or more of four low-effort signals (a changed answer on the repeated task, speeding, or identical answers across profiles in either experiment), against 8 to 13 percent of everyone else. So saying yes goes with low effort, but most of the people who said yes did not show two or more of these signals.13

How each panel handled self-reported AI use

The self-report was still used for billing. With both Bilendi and Rakuten, I agreed to replace a self-reporter if they met at least two of four criteria. These were (i) saying something different for the repeated task, (ii) being faster than half the median time, (iii) answering the same for all profiles in the first experiment, and (iv) answering the same for all profiles in the second experiment. Self-reporters who met none or only one of the criteria were kept: 79 in Germany, 150 in South Korea and 286 in Singapore. I approved all my Prolific completes (Prolific has no pool to replace them from), and I only dropped US participants for the pre-registered reasons. The attention check accounted for more replacements than the AI rule in all three countries. Replacement means the panel did not bill me for the respondent and recruited a new one in their place. I still have the data on everyone who was replaced.

This is worth knowing for anyone running a study like this. Prolific is automated, so I never discussed any of this with anyone from the company — although they seem quite responsive and if I had major concerns, I turst I would have been able to get in touch with someone involved with quality control. The other vendors were very interested in all of this and willing to look into any respondent I suspected of being low quality — for whatever reasons, AI or not. Of course, sample quality is central to their business model. You can typically work with your project manager to make sure you end up with the best-quality sample possible.

What the checks can and cannot catch

In these four online panels in the summer of 2026, I found no measurable sign of automated responding. The problem that did show up was low effort, from respondents who sped through, straightlined, or answered the repeated task differently.

So, I think I collected generally good-quality samples. After doing the replacements, I found that when I showed the same task again, respondents answered the same as the first time 78 to 80 percent of the time in Germany, South Korea, and Singapore, and 89 percent of the time in the US. This is at or above the 77 percent average in Clayton et al. None of my automation checks flagged anyone, and I only had to replace 5 to 13 percent of completes.

The checks I used I think work well to detect crude forms of automation, but they would not catch someone running an AI assistant in their browser, as the AI could generate normal browser events for all input. If the AI agent was sophisticated enough, it would see the page just as a human does and would not fill the hidden field. Mouse movement might help here for now, as someone using a computer had a median of about 250 mouse movements per screen (in US respondents answering no to my AI question), but most non-US respondents were on touchscreens. These measures only catch the kinds of automation I thought to look for. There could be others slipping through undetected.

Insofar as lessons go, I recommend using layered checks. In particular, I recommend using behavioral checks on a repeated screen, and then if a respondent answered yes to the AI question, deciding whether to replace them based on those behavioral signals.

So is the attention check compromised? Perhaps, if someone is trying to defeat it. However, I found no sign of anyone doing that in my samples. People simply rushed through the survey, and a repeated task, a clock and the attention check were enough to catch them. Whether or not my findings from my summer surveys can be relied upon going forward is a big open question. I think probably not. Browser agents are getting cheaper and better at acting like people, and the checks that came back empty here are the first ones they will learn to pass — if they aren’t already passing them. The case for running them anyway is they are easy to implement and cost you basically nothing. You can build these into, for example, a Qualtrics AI design skill, taking into account any new contributions that people have made. And perhaps in collaboration with your vendor, who has the strongest interest in the world (their business model) to ensure that their samples are human and that their survey takers are not automating the process.

Subscribe now

Subscribe now

If you annotate or scale political text, you are probably wondering whether JEV is really cheaper, faster and better calibrated than other options, and whether it gives up any accuracy to get there. To find out, Matt DiGiuseppe and I re-ran seven applications from the political science literature in a new working paper, now on arXiv (Denney and DiGiuseppe 2026). For each application, we scored JEV against the same human benchmark as the original paper, along with a current low-cost commercial model (GPT-6 Luna) and an open-weight model you can run locally (Qwen3.8-27B).2

Overall, JEV is fairly close to the other models in accuracy. It is generally a bit worse than Luna on the reading tasks but better than both models on the V-Dem codes. It is not cheaper than Luna at batch prices, but it was much faster for us. With each question asked only once, it also produced better-calibrated probabilities than Luna’s token probabilities. Compared with Qwen, it depends on the task.

Seven Text as Data Replications

We examine seven applications of annotation and scaling. In the first five, the model has access to the text itself. The last two, which we call recall tasks, give the model just a name, and it must answer solely on the basis of what it learned during training.

• tweet relevance for content moderation (Gilardi et al. 2023);

• sentiment in tweets about two Supreme Court decisions (Ornstein et al. 2025);

• attack types in 37,709 Global Terrorism Database incidents (Brandt et al. 2026);

• pairwise scaling of open-ended survey answers on economic knowledge (DiGiuseppe and Flynn 2026);

• sentence-level positions in UK party manifestos and European Parliament speeches (Le Mens and Gallego 2025);

• left–right positions of European parties from their names alone (Di Leo et al. 2025);

• V-Dem expert codes for 53 indicators in 171 countries, again from the country name alone (Weidmann et al. 2026).

To be as consistent with the original studies as possible, we used their prompt wording where JEV allowed it but left out the instructions about the format of the answer. In cases where a study had used a numeric scale, we rephrased the scale in terms of JEV’s worded answer levels. In cases where a study had allowed for a “not applicable” answer, we added a separate yes/no question instead. All of these changes are described in the appendices of our paper. I go through the questions in order, starting with accuracy, then cost and speed, and finally the probabilities.

Accuracy: close, but rarely ahead of Luna

Figure 1 shows the level of agreement that each model achieves with the human benchmark (each study uses its own measure for this). No single model outperforms all others, with JEV underperforming Luna on tweets, manifestos and EP speeches, matching it on survey pairs and party positions, and doing slightly better at identifying the type of attack. It also outperforms both models (and the published GPT-4o codings) on V-Dem. Compared with Qwen, JEV does better on attack types and both recall tasks, worse on tweet sentiment and the 2023 tweets, and about the same elsewhere.

One caveat applies to the recall tasks. Both the V-Dem data (version 14) and the expert party placements were publicly available before all three models were trained. To the extent that they were part of the training data, this is a form of data contamination, and some of the agreement may come from the models recalling the benchmark. This matters most for the comparison with GPT-4o’s V-Dem codes, since GPT-4o was trained before V-Dem version 14 was released.

Cost and speed: faster, not cheaper

At OpenAI’s Batch prices (which apply when you are willing to wait up to a day for results), Luna is between 34 and 55 percent cheaper than JEV on five of the seven tasks and within 3 percent of JEV on the other two (see Table 1). If you need your results immediately and pay the more expensive Standard prices, JEV is cheaper on six of the seven tasks.

TypeSafe’s pricing is less helpful than it first seems. TypeSafe does not charge for output, which matters for a reasoning LLM that writes many output tokens, but in a classification task the labels are so short that input makes up most of the cost. Luna wrote around four tokens for each answer, while JEV billed 1.2 to 3.3 times as many input tokens per decision, because it counts the question and all of the answer options as input. Qwen was more expensive still through a hosted API, at 2.35 to 5.41 times the cost per decision of JEV, but a 27B model is small enough that you might be able to run it on a good computer or on your university’s supercomputer (like Leiden University’s ALICE).

In our runs, JEV was much faster than the other two models (0.09 to 0.27 seconds per decision, against 0.75 to 1.04 for Luna and 0.39 to 0.87 for Qwen), although for most projects you can run a batch job and come back the next day for results.

Probabilities: easy to get, and better calibrated than Luna’s

TypeSafe’s third claim, and in my view the most interesting, is about JEV’s probabilities. Researchers typically use such probabilities in one of two ways. Some send the answers the model is unsure about to a human coder, or search a corpus for a rare category. What matters then is that the probabilities rank the right answers above the wrong ones, which AUROC measures. Others build a continuous scale from the probabilities or read them as rates. That requires the probabilities to be calibrated, meaning that the model is right 80% of the time when it gives an answer 80% probability, which the expected calibration error (ECE) measures. Chat-tuned LLMs are known to be overconfident, and many APIs give token probabilities for only a few of the top tokens, or none at all (OpenAI gives at most the top five for Luna, and none when it reasons).

Take a look at Figure 2 here. In the party-comparison task, 74% of Luna’s probabilities are smaller than 0.01 or larger than 0.99. For JEV the figure is 18%, and for Qwen 16%. If you ask a model which of two parties is more right-wing and the two are close to each other on the political spectrum, a reasonable answer is a probability somewhere in the middle of the range. JEV and Qwen give that kind of answer far more often than Luna does. Since Qwen does it too, near-certainty is not inherent to LLMs in general. Or at least not to Qwen.

Extreme probabilities are a warning sign, but they are not proof of poor calibration. To check calibration, we compare the probabilities with the human labels, and Figure 3 shows the results. It covers every task where we have human labels and probabilities from all three models, which includes the tasks we replicated and four tasks we added. On all eight reading tasks where we ask each question only once and read Luna’s probability from its answer token, Luna has a higher calibration error than JEV. When we ask each pair in both orders and average, the gap narrows, and on the survey pairs it disappears. On the task where Luna writes out its probability (attack type), Luna is better calibrated than JEV.

Against Qwen, each model is better calibrated on its own kind of task, and by quite a lot. Qwen is better calibrated on all four pairwise tasks. JEV is better calibrated on three of the four labelling tasks, on attack type and on both recall tasks. On V-Dem, for example, JEV’s calibration error is 0.06, while Qwen’s is 0.15.

The ranking results point the same way. Compared with JEV, Luna’s AUROC is lower on seven of the eight tasks where each question was asked once, and equal on the other. Compared with Qwen, it is lower on all eight. JEV and Qwen are not consistently different from each other.

Subscribe now

JEV! Should you use it?

JEV does not save you money if you can wait for batch results, and it is not more accurate than Luna on most reading tasks. On the other hand, it is very fast, and unlike Luna it gives you a clean probability for each answer, rather than making you infer it from a handful of tokens that often have probabilities close to 0 or 1. For most social science work, the probabilities are the more useful of the two. The speed difference is big, but it does not matter unless you need the answers immediately, as in market research, rather than a day later from a batch job.

Qwen is at least as accurate as JEV on most reading tasks and better calibrated on the pairwise ones. If you are able to run an open-weight model yourself, for example on a university cluster, you may not need JEV. JEV is ahead of Qwen on the recall tasks and on attack type, though, in both accuracy and calibration.

TypeSafe has not released any of JEV’s internals (its architecture, training data or reward), so there is little to judge it on beyond how it performs on tasks like these. As with any LLM, that means validating it against human coding for the task you are interested in.

***

The full paper, with all seven replications, the prompts and the appendices, is on arXiv: Denney, Steven, and Matthew DiGiuseppe. 2026. “JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications.” Preprint, arXiv, October 5. https://doi.org/10.48550/arXiv.2610.06625

Most of my work with a command-line agent is writing prompts. The agent (whether it’s Claude Code or Codex) does best when I give it as much information as possible. It’s really good at figuring out what I want from the latent constructs in the text I give it. That means my prompts can easily run to 300 words (or more), and typing 300-word prompts all day is energy-sapping. It’s drudgery. So the rational move is to make the prompt shorter, which degrades the answer and the work being done.

Subscribe now

Earlier this year I realized how good speech-to-text has become, and I set it up on my desktop (Linux) and my MacBook Pro. Almost all my prompts are dictated now. It works like this: I hit a hotkey, talk into my mic for as long as the thought takes, hit the hotkey again, and about a second later the cleaned-up text is sitting at my cursor. My prompts are much longer as a result, and the output is better, because a language model can work out what I want even if I ramble a bit. It needs lots and lots of info and detail for it to work properly, and speaking is the cheapest way to give it that. I still type when I need exact wording though, like slash commands, paths, flag names, model IDs, etc. Otherwise, it’s all spoken.

I find this really useful when I need to explain a lot of stuff and closely supervise the work, like iterating on my own writing, drafting emails, turning rough ideas into prose, or providing detailed feedback on student assignments, especially writing assignments. When providing feedback, the bottleneck is often how much I’m willing to type out. Feedback requires a lot of explanation: what the problem is, why it is a problem, how to revise, which of multiple revision options is better, etc.

It’s not just a matter of talking being faster than typing; it’s also that I can think more fully when I talk. I can talk through an argument, qualify a judgment, add examples, revise as I go, and then clean it all up by editing (mostly by typing, but some by talking, too). This has made my advising of students a lot better, allowing for more specific, more useful feedback. I haven’t run an A/B test, but my impression, and I tell my students this, is that the feedback is much better than what they’re used to getting.

Typing isn’t obsolete, though. It’s still better for anything that has to be exact, and the best approach uses both. Your words also get modified somewhat on the way to text, especially if you want clean output; more on that below. In this post I cover how much the technology has improved and then the setup I run on Linux and macOS. Both are public: hyperwhspr for Linux and macwhspr for the Mac. Speech-to-text has lots of uses, but what follows comes mostly from my experience working with agentic AI.

Speech recognition is good now

For a long time, dictation was frustrating because fixing the errors took longer than just typing would have. In 2011, researchers at Microsoft cut the word error rate on conversational telephone speech from 27.4 percent to 18.5 percent by replacing Gaussian mixture models with deep networks (Seide, Li, and Yu 2011). This may sound pretty bad (almost one word in five is still wrong), but at the time it was a huge breakthrough. By 2016 the error on the same benchmark was 5.8 percent, on par with the then-measured 5.9 percent for professional human transcribers (Xiong et al. 2016). In 2017 it was 5.1 percent (Xiong et al. 2017).

OpenAI’s Whisper model made high-quality speech recognition accessible to all. The model, trained on 680K hours of data, achieves a zero-shot error rate of 2.7 percent on the LibriSpeech benchmark. It also shows a big improvement in robustness to accented speech and background noise compared to previous models (Radford et al. 2022). In 2023 OpenAI put it behind an API at $0.006 USD per minute (OpenAI 2023), or 36 cents per hour of speech. The 2025 generation, gpt-4o-transcribe, beats Whisper across the FLEURS benchmark, by OpenAI’s own evaluation (OpenAI 2025). Note: these numbers are all from different test sets so are not directly comparable, but they give a sense of the improvements in quality and cost.

The main remaining issue with speech recognition, once you get the accuracy good enough, is the latency after you stop talking before you get the transcription. The 2026 releases handle this by streaming the transcription back to you as you’re talking. ElevenLabs claims it can do this in under 150ms with their Scribe v2 Realtime model (ElevenLabs 2025), and OpenAI does it with gpt-realtime-whisper, which streams over a WebSocket, for a cost of $0.017 per minute (OpenAI docs). I just switched my two machines over to use it this week and the improvement is great.

The setup

The pipeline is the same on both machines. I hit a hotkey to start recording. As I talk, the audio is streamed to a transcription model. Then I tap the hotkey again to commit the buffer and, after about a second, I get the final transcript. I run it through a small LLM to clean it up and then paste it at the cursor, which is usually in a terminal, but sometimes in a draft like this one.

hyperwhspr, for Linux

On Linux the recorder is hyprwhspr, and scdenney/hyperwhspr is the pattern I use to reproduce my configuration under Omarchy/Hyprland. It also includes the config for the backend to make it stream, the code for the hook that calls the cleanup model with a constrained prompt, the vocab file I’m using, a systemd service file (with some Wayland fixes), and the command I use to calibrate (feeding the cleanup model’s log in to create new vocab rules). I have a README with a dated table of all the issues I know about (the latest being this week’s move to a streaming model). The items are added either by my agents or by me.

macwhspr, for the Mac

There’s no hyprwhspr for macOS, so scdenney/macwhspr is the whole thing. It contains an idempotent install script and a small Python daemon that uses SoX to record audio and stream it to the transcription service, with the Globe key (press and release) to start and stop recording, and Karabiner and Hammerspoon for the hotkey and status indicator. The daemon runs in the background via launchd. If your transcript lands in the wrong window, you can recover one of your last 20 recordings with Ctrl-Cmd-V. Apart from the platform-specific bits, it uses the same transcription, cleanup, calibration, and logging pipeline as the Linux setup.

There’s one piece I initially underestimated, and that’s the cleanup stage. The transcription model inevitably produces messy output, with filler words, false starts, and little punctuation. To clean it up, I feed it into GPT-4.1 mini, very carefully prompting it to only add punctuation, paragraphs, and other formatting, and not to change the meaning of the transcription in any way. This way, if I prompt Claude by asking it a question, it will receive a properly formatted question, not an answer to my question. I also log all cleanups, and have a calibration command that goes through the log and updates a vocabulary file of spellings, names, romanizations, and other common corrections I want the cleanup stage to make. This is the main way the setup is personalized, and it gets better with usage.

Below is an example of what the setup looks like in action on my Linux machine. I press a hotkey to start, a pill appears that updates in real time as I speak, and when I press the hotkey again, the cleaned-up output is fed into the Claude Code prompt.

Latency differences

Previously, for both machines, the upload happened only after I had finished talking, and the time taken depended on how long I spoke for. Now that I have switched to the streaming backend, the work is done while I am still talking.

I tested this with the same audio (synthetic speech streamed at real-time pace over the same network), the only change being the backend. It’s a pretty modest difference for the short (6.5 second) clip (1.7 seconds for batch versus 0.9 for streaming), but for the longer (94 second) one, it’s an enormous improvement (5.9 seconds for batch versus 0.9 for streaming). Across the lengths I tested, streaming consistently takes about a second, regardless of length. My most recent dictation in the log is 128 seconds long (I was complaining about how LaTeX was laying out a table in a manuscript, undoubtedly an experience familiar to many of you); with batch I had to wait 5 seconds just for the transcription to appear. Now I wait about 1. Given that I do dozens of these per day, this is a big improvement. The new model is much better.

Each cell in the above chart corresponds to a single run. Similar results are visible in the production logs. The downside is that streaming is more expensive (0.017 USD/min vs. 0.006 USD/min). For a two-minute dictation, this translates into ~0.03 USD. Other minor downsides: (1) the realtime model does not accept a vocabulary prompt, so domain-specific terms have to be dealt with in the cleanup stage, and (2) websocket communication introduces additional failure modes. To mitigate (1) and (2), both repos allow you to switch back to the batch model with a one-line config change. Finally, if you do not want your audio data to leave your machine, hyprwhspr offers a locally-hosted ONNX backend that requires ~5.7 GB of VRAM to run, and you have to maintain the model yourself. I found it usable, but IMO the quality isn’t good enough to justify the hassle.

Subscribe now

What a month of this costs

People often ask how much this costs to run, so before writing this post I went back and grabbed my logs from the last 30 days (June 13th to July 13th) on both of my machines. It turns out I’ve invoked dictation 662 times for a total of 7.7 hours of dictation, generating 54,796 words. The cost to transcribe those words was $2.77 USD; the cost to run the cleanup model was another $0.27, for a total of $3.04, or about 5.5 cents per thousand words. Keep in mind that this is using the old, batch-oriented backend. If I had been using the streaming backend (which is what I’m using now), it would have cost about $8.11 for the month. The difference is totally negligible for me, and really the only thing that matters is the latency.

Make the switch

I should mention that typing is still useful and will continue to be used when exactness is important, such as commands, file paths, quoting text, citations and other identifiers. Also, I still do my writing by typing. I’m not going to dictate a scientific manuscript, at least not anytime soon.

But typing shouldn’t be the only way you work with an AI agent. Only typing creates a trade-off where you either under-contextualize the agent or spend more time and energy contextualizing it. A combination of the two works best. Explain the problem to the agent out loud (the transcription and cleanup give you a first draft, which you can always edit), then type out the specific commands, paths, etc.

Voice input has reached a level of accuracy, cheapness, and immediacy where it’s no longer a gimmick or just an accessibility tool. If you’re doing serious work with agents and still typing out all your prompts, you’re bottlenecking yourself. If you’re not already using speech recognition to give detailed instructions to the agents in your computational (or non-computational) pipelines, you should be.

Note: Data and replication code for the analysis provided here are available on GitHub.

The United States turns 250 this year, and the old argument over what it means to be American remains as intense as ever. What better moment than now to engage in that age-old debate of what it means to be an American? Flatten by media coverage, the debate usually gets cast as two options. Either the country is a creed, a set of ideas anyone can sign up to, or it is a heritage, as a matter of belonging that you inherit by birth and/or blood.

That two-sided framing is not merely an artifact of punditry. It dates back at least to the historian Hans Kohn, who, in 1944, distinguished between civic and ethnic nationalism. Civic nationalism defines the nation in political terms, as a community of citizens bound by shared institutions and ideals. Ethnic nationalism defines it in cultural and genealogical terms, as a people united by common ancestry, language, or heritage. At its core, Kohn's distinction concerned how people answered a fundamental question: who belongs to the “circle of we"? To a large degree, this civic-ethnic distinction has organized the study of nationalism ever since. Scholars have, of course, challenged how clean the divide really is. As Anthony Smith argued, even civic nations draw on ethnic myths and symbols.1

Subscribe now

As public opinion surveys became a cheaper and more reliable way to measure public opinion, political scientists and sociologists began asking people directly who belongs to the nation and how they relate to it, rather than inferring those views from elite rhetoric. Deborah Schildkraut’s Americanism in the Twenty-First Century and Elizabeth Theiss-Morse’s Who Counts as an American? both show that Americans’ views are more varied than a simple creed-or-heritage divide suggests.2

When Americans are asked directly, they do not divide neatly into two camps. Using a method that identifies recurring patterns across attitudes toward national belonging and national pride, Bart Bonikowski and Paul DiMaggio distinguish four dispositions. Their typology, however, was developed from a 2004 survey and only later compared with data from 1996 and 2012. The question is whether those same four types still structure American national identity today.

I replicated their approach using the International Social Survey Programme’s (ISSP) National Identity module, which asked the same battery of questions in 1995, 2003, 2013, and 2023. I find, again, the same four identiy types. Americans do not sort into two competing visions of the nation, but four. What has changed over the past three decades is the relative size of those groups. One has remained remarkably stable, while the others have risen and fallen.

Two questions, four Americas

The types follow from two basic categories of questions. First, who counts as truly American? Is it about being born here and being Christian, or about citizenship and respecting the country's institutions? Second, how warm you feel about America. How proud are you of the country? Crossing these two dimensions gives you four identity types.

The horizontal axis captures who counts as truly American, from inclusive (citizenship and shared institutions) to exclusive (birth and religion). The vertical axis captures national pride. The types and their attitudes are then as follows:

• Ardent: “hot” nationalists, scoring high on everything. They want high boundaries for belonging and are proud across the board.

• Restrictive: people who also draw distinctive boundaries of belonging and are sure the country is superior to others, but their pride in its actual achievements runs cooler than ardent nationalists.

• Creedal: those who are proud and inclusive. They anchor belonging in citizenship and institutions rather than birth or ethnicity/religion, and they are less likely than ardent or restrictive nationalists to say the country is better than others.

• Disengaged: those “cool” on all of it, rejecting or downplaying any sort of national identification.

You can see each type's profile in the survey responses. The clearest divide is not national pride, which most engaged Americans share, but the boundaries of belonging. The ardent and restrictive types are much more likely to say that being Christian is part of being truly American and that the United States is superior to other countries. Creedal Americans reject both of these items as conditions for national membership. They are low on the ascriptive criteria (born here, be Christian) and on superiority, but high on the civic criteria (citizenship, respecting institutions) and moderately proud, the signature of an inclusive-but-attached patriotism. The data presented so far already complicates the creedal-versus-heritage framing.

How the types are measured

Since this Substack is meant to be somewhat technical, a word on how the types are constructed. The data come from the ISSP National Identity module, which fielded comparable nationally representative U.S. surveys in 1995, 2003, 2013, and 2023. I use the sixteen items included in all four waves.

As the figure above shows, the questions fall into three broad domains: who counts as truly American, how proud respondents are of different aspects of the country, and whether they express a sense of national superiority or unconditional loyalty. The first domain distinguishes civic criteria, such as citizenship and respect for institutions, from ascriptive ones, such as birthplace and Christianity. The first eleven items use four-point response scales; the remaining five use five-point agree-disagree scales.

The identity types are derived from latent class analysis. The method treats the population as a mixture of unobserved groups, each defined by a distinct pattern of responses. It estimates both the size of each group and the probability that each respondent belongs to it. Unlike factor analysis, a commonly used alternative in the study of national identity, which places respondents on one or more continuous dimensions, latent class analysis identifies qualitatively distinct types of people whose responses cluster together. That makes it well suited to studying national identity when the question is not how much nationalism people express, but how they combine different ideas about belonging and national pride. Bonikowski and DiMaggio adopted this approach, which is why "creedal" here refers to a type of respondent rather than a high score on, say, a creedal scale.

I estimate latent class models with one to six classes and selected the four-class solution based on model fit and interpretability. The classes are well separated, and the same four-type structure appears in all four survey waves.3

What changed

The next question is how these four conceptions of the nation have changed over time. Below is the proportions belonging to one of the four over the last thirty years. What do we see?

Two things stand out. First, the two hot types have receded. The September 11 terrorist attacks produced a sharp but temporary surge in more exclusionary forms of nationalism. The ardent type reached its peak in 2003 before steadily declining, while the restrictive type has fallen below its pre-9/11 level. Second, the creedal type experienced the inverse of that surge. Its share fell sharply in 2003 but returned to its long-run level of roughly one-third of Americans by 2013, where it has remained since. Across three of the four survey waves, the inclusive conception of the nation is the modal form of American nationalism.

Perhaps most striking is the growth of the disengaged. In 1995, they accounted for about one in eight Americans. By 2023, they had become the largest of the four groups, comprising more than one-third of the population. Because these are repeated cross-sectional surveys rather than panel data, we cannot conclude that individuals moved from the ardent or restrictive types into disengagement. But the aggregate pattern is consistent with exactly that possibility.

This finding should hold our attention. In 2023, the most common way of relating to the nation is detachment. It is hard not to make this connection to a broader retreat from shared life and experiences. Interpersonal trust has fallen, while community and religious membership has thinned out. Isolation has risen to the point that the Surgeon General has called it an epidemic. The problem is less that Americans hate one another's idea of the country than that a growing number are opting out of having one at all.

Politically, the next obvious question is whether this shift has unfolded similarly between left/right partisans. Looking separately at Democrats and Republicans show both encouraging similarities and notable differences.

The most striking finding is the stability of the creedal type across the partisan divide. Throughout the past three decades, roughly one-third of both Democrats and Republicans have held an inclusive but proud conception of the nation.

Where the parties differ is in the trajectory of the other three types. Among Democrats, the hot and boundary-drawing types declined, with most of that shift moving toward disengagement, which now accounts for more than four in ten Democrats.

Among Republicans, the restrictive type remained relatively stable, although it now appears to be declining as well. Instead, the defining change over the past three decades was the sharp post-9/11 surge in ardent nationalism, followed by its steady decline. Disengaged Republicans also became more common, although they remain much less prevalent than among Democrats.

The case for liberal nationalism

The creedal conception of the nation has proven more durable than the others. For those who see liberal nationalism as the most desirable basis for democratic politics, that is encouraging news. Creedal Americans define national belonging in terms of citizenship and shared institutions rather than ancestry, while expressing pride in their country without believing it is superior to others. This is what political theorists call liberal nationalism. It is, among self-identifying nationalists, a normatively desirable form of thinking about community in the nation-state era.

The central claim of liberal nationalism is straightforward. A shared civic identity can sustain liberal institutions by encouraging strangers to see one another as members of the same political community. Rather than defining the nation by ethnicity or religion, it defines membership through citizenship and shared democratic institutions.

The distinction is not merely theoretical. It also lies at the heart of one of the country’s most consequential constitutional debates: birthright citizenship. Creedal Americans place relatively little weight on being born in the United States as a condition of being “truly American,” but they overwhelmingly regard citizenship as essential. The logic of the creedal conception therefore points toward jus soli — citizenship, rather than ancestry, is the basis of national belonging. The Supreme Court’s recent decision In Trump v. Barbara (2026), reaffirming birthright citizenship under the Fourteenth Amendment, reflects this civic understanding of national membership.4

In recent years, a growing number of scholars have argued that liberals should reclaim this inclusive form of nationalism rather than cede the idea of the nation to more exclusionary visions. Writers making that case for America, from Noah Smith to Alexander Kustov and others, argue that the strongest foundation for national identity is one rooted in citizenship rather than ancestry, open to newcomers, and confident enough in its ideals to welcome them as fellow citizens. Attempts to bypass national identity altogether, whether through cosmopolitanism or other forms of supra- or non-national identification, they argue, simply leave the language of nationhood to the demagogues.

My findings suggest that this civic conception of the nation is more than a normative ideal. While ardent, restrictive, and disengaged forms of national identity have expanded and contracted over the past three decades, the creedal type has remained remarkably stable, accounting for roughly one-third of Americans in every survey. Nor is that durability merely an American curiosity. In my comparative study of twenty-nine countries, the United States emerges as the clearest example of civic nationalism: the country where commitment to democratic rights most strongly constrains hostility toward outsiders.5

Subscribe now

The road ahead

So the mood on America's 250th should be one of cautious optimism. The meaning of the American nation remains unsettled. The “bets still on the table”. The civic conception of the country has endured, exclusionary nationalism has not become dominant, but a growing share of Americans remains disengaged rather than committed to either vision.

There is, to be sure, a real loss of attachment. But that also leaves space for persuasion. As Ross Douthat argues, America’s enduring strength is its capacity to continually renew itself. This is a contested nation, but diversity is somethig most Americans think makes the country stronger, anyway.

The central challenge, then, is not simply to contain exclusionary nationalism, which is always going to exist in some form or another. It is to build a more compelling alternative. The evidence suggests that the most durable conception of the American nation is also the most small-l liberal. It is the one that grounds belonging in citizenship, shared institutions, and common political commitments rather than ascriptive characteristics. For advocates of liberal nationalism, the task is therefore not only to oppose exclusion, but to persuade the growing number of disengaged Americans, especially on the left, that the American experiment is still a project worth identifying with and investing in.

Earlier this month, a thread on X that went quite viral pointed out that a prominent political scientist had a half-dozen fabricated citations in a recent article on South Korean politics.¹

The reaction was about what you would expect: disappointment, mostly, and in some quarters shock. No instructor would accept this from a student, and when it appears in a published article it has to be addressed, through a correction or a retraction. Given the number of bad references, I imagine most people following the case expect the latter.

Subscribe now

Still, the speed and the intensity of the response are worth pausing over. A case like this should have been handled with more care.

First of all, we do not know how the citations got there. It is unlikely, but it is possible that the fault lies with the journal rather than the author. Many of us have concluded that this outlet does not meet the standards it advertises, and at the time it was turning articles around very quickly to get thoughts out on the fast-moving events in Korea. It would not be out of the question that an editor ran the article through an AI program to tidy or reformat the references, and that the program made them up. I do not know whether this is what happened, and it seems like this information should have come out by now. But neither does anyone else.

It is fair to call out errors of this kind. But it is also important to be certain that the grounds for a public accusation of malpractice against an individual are solid, because once it is made, it cannot easily be undone. The damage to reputation is immediate and severe, and a later correction, if one comes at all, rarely gets the same attention.

It is also a time for humility. In the most mundane and boring parts of academic work, such as locating a source and formatting a reference, many of us have had the rug pulled out from under us. These tools are very good at making it seem like the work is done correctly when it is not, and they can trick even otherwise careful people into thinking that their personal robot got it right.

I suspect that much of this anger is coming from a broader anxiety about what AI is doing to the academy. All that to say, we should be far more sure of what happened before we make an example of someone who has, by and large, done excellent work, and we should consider the case in the context of larger structural issues.

To be clear, I do not actually know how this happened. But I am far more inclined to believe that it was a case of misplaced confidence in a technology that routinely spits out convincing falsehoods than a case of calculated fraud. That does not excuse the error, but it does change what we should learn from it. The failure may be less an individual lapse than a structural one: a culture that has adopted powerful tools without equally powerful norms of verification. The good news is that this is a problem we can fix — and I have some solutions to try out. But first, we need to understand how we got here, and why the machine is so good at deceiving us.

Old habits die hard

Type a scholar’s name and a couple of keywords into Google Scholar, and you get a list of citations. Use a few of them without reading past the abstract because you already know roughly what that person argues, and you have done one of the more mundane tasks of writing. AI is just exposing an inconvenient truth. We do not (closely) read every source we cite. Some citations are indeed used as evidence, and some are presented as a kind of performance. They signal that we know the literature or that we have cited whoever a reviewer expected or explicitly asked us to. A chatbot like ChatGPT or Claude, in these early days, feels like the same exercise. You ask for the literature on a subject or particular claim, and it gives you names you recognize along with titles that look right. The problem is that some of those titles do not exist.

None of this is new. Take one prominent example. A decade ago, before anyone could blame a model, a prize-winning history of North Korea was found to cite nonexistent or irrelevant sources in at least sixty-one places. The author returned the prize and left his chair.2 A milder form of the same problem is extremely common: a wrong DOI, a page range off by two, a citation to a real paper for a claim it does not make. These are hallucinations in all but name. What AI adds is ease of use and an appearance of accuracy, and that is one of the big structural issues we are dealing with now.

Why the machine makes things up

Google Scholar (usually) returns sources that actually exist. A large language model returns what is probable. It is not a database, nor a search engine. It spits out the next most likely token given everything before it, so when you ask for a citation, it produces a string that looks like one: an author, a year, a title, a journal, and a volume, because that shape is overwhelmingly probable wherever citations appear. Nothing in that process checks the reference against a master list. The model does not care whether the work exists, which makes it a bullshit machine in Harry Frankfurt’s sense of the term. This problem is deeply entrenched in these models, as the authors of a recent paper from OpenAI point out. When we train and evaluate these models, we usually end up rewarding a confident guess over an admission that the model does not know the answer.3

How often does this happen? In one controlled study, GPT-3.5 fabricated 55 percent of the citations it generated for literature reviews. The newer GPT-4 was much better, but still fabricated 18 percent.4 In an audit of two and a half million biomedical papers, the rate of fabricated references rose more than twelvefold after 2023, to about 57 per 10,000 papers by early 2026. The steepest rise came with the introduction of new AI writing tools in mid-2024. Most of these references had made it through peer review, which is understandable, since it would be a lot of work for reviewers to check every source.5 The problem reaches well beyond the academy. In 2023, a federal judge fined two New York lawyers five thousand dollars for submitting a brief that included six judicial opinions ChatGPT had invented. One of the lawyers had used ChatGPT to verify that one of the opinions was real.

We do not know how the fabricated citations in the case described at the start of this post were produced, but my guess is the simplest one: someone used a single prompt to a general-purpose chatbot, with no tools or documents to check against. That is the weakest possible way to use the new technology, and it almost guarantees some hallucination in the answer(s). Tools that retrieve and read sources before answering do much better, but they still have a way to go. When Stanford tested retrieval-based legal tools that vendors marketed as “hallucination-free,” they still gave a false answer in something like one out of six cases. But this is at least a sign of things moving in the right direction.

At the very least, we need verification systems that can find hallucinations and other errors.

Responsible all the same

We are accountable for what we publish. The best way to avoid this problem is to read the sources we use carefully, check every citation, and leave out anything we cannot verify. Fake sources are unacceptable whether they come from misusing an LLM, a tired research assistant, or our own carelessness in formatting.

For students, that standard has to be explicit, and at my institution it is. I am not impressed by how universities (mine included) mostly cast AI as a tool for cheating, but Leiden University’s Faculty of Humanities gets the definition of fraud right: any act or omission that prevents a correct assessment of a student’s knowledge. Its examples include the unauthorized use of AI software and the invention of research data.6 Its guidance on generative AI is more direct: “GenAI generates language, not information … If asked to produce references, many LLMs will simply invent non-existent citations.” A fabricated citation misrepresents the evidence the work rests on, which is exactly what assessment is meant to evaluate. So yes, if taken literally, it is fraud, and the heaviest penalty on the books, exclusion from all examinations for a year, should be deterrent enough.

A field-wide response is going to be harder than a local one, and the distance between what these policies aim to resolve and what actually happens seems quite wide. It seems like people are trying to police their way out of this problem. That comes from a good place, a wish to make sure people can do good science, and it is understandable. In mid-May, for example, arXiv announced that authors who post papers with hallucinated references will be banned for a year, and after that can only post papers that have cleared peer review elsewhere.7 I like the motivation, but not the mechanism. The policy is, as has been noted, “welcome but unenforceable.”

The early evidence backs this up. When one peer-review service ran its reference checker over two hundred papers posted to arXiv after the ban, more than a quarter of them still included a hallucinated citation. Despite the threats of punishment, it is obvious that people are using AI as a research tool. The technology is too good and too useful for prohibition to hold. People are going to use it to think with and write with, whatever the rules say. To the extent that the deterrent messaging is heard at all, I think it is mostly leading to underreporting of AI use because of social desirability.

We need a different approach. The answer is to teach people to use the tools openly rather than in secret, and to turn the same tools on the problems they have made worse. We can, for example, build them into the checking process when someone submits work to a journal. We can also use them to help writers, senior authors and students alike, audit their reference lists and check that each source actually supports the claim attached to it, until this becomes routine. Done properly, this checking could be more systematic and reliable than it ever was before ChatGPT. Rather than tell people what they cannot do, we should show them what they can do instead.

Skills / chatbots

The first thing we need to do is stop using chatbots as the default for serious research and writing, and start building (or using) skills for agentic AI instead.

A skill is a set of instructions that an agentic AI system uses to perform a task. A skill constrains what the AI can do by giving it specific steps to follow and the tools to use for those steps. It also gives the model steps to check its own work. With a one-shot prompt, we are asking the AI to be creative. With a skill, we are taking that opportunity away. A skill can tell the model to dispatch agents, to look up sources in actual indexes, and to tell the difference between what it confirmed and what it merely guessed. Fabrication comes from a model answering from probability instead of evidence, and a well-built skill helps close off that option.

I write these for my own research and keep them in a small library, the Open Science Skills. Three of them are relevant here.

The first is a reference checker (citation-check). This one most directly addresses the problem discussed in this post. It reads every in-text citation and every bibliography entry in a paper and then checks each reference. It confirms that the reference exists by looking it up in programmatic indexes such as Crossref and OpenAlex, not in the model’s memory. For references with a DOI, it checks that the DOI resolves to the cited paper and not merely to some paper. If a title looks made up, it checks it against the named author’s actual body of work. Only if it cannot find the reference by title or by author does it call it fabricated. What comes back separates the serious problems from the cosmetic ones and flags the uncertain cases for a human. It is not perfect, but overall, it works.

The second skill addresses a different, arguably much deeper problem, one that is much less likely to go viral on X. Even if a citation is real and formatted correctly, what if it does not actually support the claim it is attached to? The source-claim checker (fact-check) reads the cited source and asks whether it supports the specific sentence, and it flags overclaiming and direction errors. To do that, it needs the sources to read. As I’ve written the skill, it cannot just look things up on the internet or “get creative.” That is what the third skill is for.

The third, a knowledge-base scaffold (research-repo), builds a project around its source library. The library has two folders: one for the original source documents (PDFs, etc.) and one for the machine-readable versions (preferably in markdown) that the checker actually reads. Each document is keyed to an entry in the bibliography, so that each source is directly connected to its bibliographic metadata. The claim checker will not run against an empty or half-built library, because to verify that you have represented a source faithfully, you need to have that source, or, where the source is not available, a file that faithfully summarizes it.

Try it yourself

In practice, none of this is as elaborate as it might sound, and the best way to figure it out is to do it yourself. For this post I put up a small, self-contained walk-through on a synthetic manuscript seeded with deliberate errors, so you can watch the checks fire. It is the first of a series of demos I am preparing for students and colleagues at Leiden University in a teaching hub (still a working prototype), with the setup guide and slides alongside it.

We are in an odd spot right now. The technology is too useful to ignore, but still unreliable enough to embarrass or even discredit us. And we have not been at it long enough to establish good habits. That is a dangerous combination.

What we need is not a moratorium, or a rejection of the technology, or a hunt on social media for the next person to make an example of. What we need is humility about how easily these tools can fool us, some grace for those who have misused them, and a serious tightening of the standards for how we use them. These positions are not inconsistent. We can hold all three at once.

Subscribe now

Correction: It has since been clarified that the article published in Korea Observer was an "annual review" reflection and was therefore not subject to peer review, which explains its quick publication timeline.

OCR has changed substantially in the last few years. For clean, modern, printed English, the task no longer usually requires a dedicated OCR engine, a domain-specific training set, or extensive preprocessing. Current vision-language models can often transcribe such material directly from a prompt, sometimes zero-shot and sometimes with a single example. In that limited sense, printed Latin-script OCR for high-resource, modern material has become a largely infrastructural problem rather than a central research challenge.

Historical documents are still harder. Old paper, damaged scans, unusual layouts, handwriting, and older typefaces all push modern models well outside their comfort zone. But even here the progress has been striking. Platforms like Transkribus routinely read European manuscripts that would have seemed hopeless a decade ago, and projects such as Loghi report similarly strong results on historical print. OCR has become much more accessible.

Korean, and by extension CJK1 and other non-Western scripts, is a different case, and early-twentieth-century Korean is different again. Pre-1933, Korean was written in an orthography that has since fallen out of use. It spelled words closer to how they sounded than to a fixed stem, and it kept letters the modern keyboard has dropped, among them the dot vowel arae-a (ㆍ) and a set of old consonant clusters. The spelling we read today is the Unified Orthography of 1933 (한글 맞춤법 통일안), the morphophonemic standard carried by Chu Sigyŏng’s lineage in the Korean Language Society, and it became the convention only after a long argument that ran through the colonial period and resurfaced in the spelling crisis of the early 1950s.2

Modern Korean OCR remains less stable than the English and European-script cases above. On the Korean portion of the PM⁴Bench benchmark the leading system changes with each new model release, and in the current version Gemini 3 Pro ranks ahead of the newest open model, Qwen3-VL, which does not improve on its own predecessor for this task. This should be read as a relative ranking within a weak field rather than as evidence of reliable transcription. The absolute scores remain low.

In one of my projects, I need OCR labels for about ninety-four million individual character images. Because I study the appearance of historical type rather than the content of the text, it is important that older Korean characters are transcribed as they were printed, rather than silently modernized. This post explores how well current models actually do.

A few tests pages

The page is from Sonyŏn (소년), a children’s magazine published in 1908, and contains a chain-rhyme word game. It is printed in vertical columns read from right to left, combining (Old-)Korean Hangul and Hanja on the same page. Material like this has traditionally been difficult for OCR systems.

The part I cared about was not whether the models could produce readable Korean. (Though even this would be helpful for many historians.) It was whether they would OCR Korean letters as printed. Before the 1933 spelling reform, Korean used forms that later disappeared or changed in ordinary writing. These include the dot-vowel arae-a (ㆍ) and several old consonant clusters. A model can read a page fluently and still normalize those forms into modern Korean. For most readers the output still looks right. For my purposes, it is wrong.

The first page was a narrow test. It happened to contain two old cluster forms, ᄭᅡ and ᄯᅱ. A modern transcription would normally write these as 까 and 뛰. GPT-5.5 kept all six occurrences of those old clusters on every run. The previous GPT model had modernized all six. That was already a meaningful change, but it did not prove much beyond this particular page.

So I tried five more pages from Sonyŏn, using prose and dialogue rather than word games. These pages contained a wider range of old Korean forms: arae-a in forms such as ᄒᆞ and ᄂᆞᆫ, several ㅅ-cluster groups such as ㅺ, ㅼ, and ㅽ, and even the rarer triple cluster ㅴ. This was the real test. The question was no longer whether a model could preserve two repeated forms on one page, but whether it could keep a broader old-orthography repertoire across different pages.

The current models can preserve old Korean forms, but not with equal reliability. Preservation is still something to check, not something to assume. In practice I treated an old form as trustworthy only when different models produced it, because a single model can also hallucinate plausible old-looking Korean.

The dedicated OCR engines were not trained on any form of old-Korean and not surprising that they performed the poorest. Mistral OCR, DeepSeek-OCR, Baidu’s Unlimited-OCR, GLM-OCR, all read the page and then modernized the old letters or fell apart, one of them looping a few character lines a few hundred times, another reading the Hangul as Chinese and drifting into a gibberish. Qwen3-VL was different. It read the Korean fluently, but silently regularized every older form into modern Korean. If your goal is simply to recover the text, that is a respectable result. If you care about preserving historical orthography, it is not.

Going back further, to 1896

After Sonyŏn, I wanted to know how the models handled even earlier Korean print. As a final test, I tried the front page of the Tongnip Sinmun (독립신문, The Independent) from 1896, the first privately published Korean newspaper, set entirely in Hangul in the full old orthography with arae-a in almost every word. The only scan I could find is small, a single 960-pixel page, which makes it a hard test on top of an old one. For once there was something solid to check it against. The Korean Wikisource was manually transcribed, so this is the one page in the whole exercise I could measure against ground truth rather than judge by eye. It should also be noted that this newspaper image is denser and has a more difficult layout then the previous magazine had.

Most models struggled. Claude Opus 4.8 transcribed only about a sixth of the page, even reversing the masthead. Qwen3-VL reached roughly two thirds, but regularized the spelling throughout. Both ended up at around ninety percent character error against the Wikisource transcription.

Gemini 3.1 Pro was the clear exception. It read essentially the entire page, from the masthead through the advertisements and price table to the end of the editorial, about ninety-six percent of the page by length. Compared against the Wikisource transcription, it reached roughly seventeen percent character error while preserving 176 of the page’s 182 arae-a. On a 960-pixel scan of an 1896 newspaper, that amounts to a complete transcription of a full old-orthography page.

GPT-5.5 failed for a different reason. It did not misread the page so much as fail to read it at all. Tongnip Sinmun is one of the most famous newspapers in Korean history, and its first issue has been reproduced countless times. When shown the thirteenth issue, GPT-5.5 instead reproduced the first: its date, issue number, and opening lines, at both image resolutions. Against the actual page, the result is about eighty percent character error. It is a useful reminder that, for well-known historical documents, a model can sometimes retrieve what it already knows instead of reading what is actually in front of it.

Subscribe now

Why this matters to someone working with the sources

The interesting part is not the benchmark. It is what happens to the historical record.

When the National Institute of Korean History transcribes historical print, it modernizes the spelling. I checked the roughly six hundred thousand colonial-era records they have published, and almost none preserve the older letters. For anyone searching the archive, the original orthography has effectively disappeared.

That makes preservation more than a technical curiosity. A model that keeps the old letters is not simply producing another transcription. It is recovering information that was lost during digitization. For anyone working with pre-1933 Korean print, this may be of importance.

The preservation results only matter if the models can also read the page accurately. On that front, the picture is encouraging. Against four pages of the 1921 magazine Kaebyŏk, where a modern transcription exists, the best current models matched roughly ninety-seven percent of the printed characters directly from the low-resolution scans. That measures reading rather than preservation, since the 1921 transcription already modernizes the spelling. Taken together, though, the results point in the same direction. Current models can read these pages, and on sufficiently old material, the best of them also preserve what makes them historically distinctive.

A note on scale

For a single page, I would use a frontier vision-language model every time.

My own project is different. It involves millions of character crops. Running a frontier model over that volume would incur substantial API costs.

Instead, I built a small voting system with more traditional OCR engines. EasyOCR and a lightweight classifier trained on historical Korean type recognize the Old-Korean, PaddleOCR reads the Hanja, and a simple voting rule chooses between them. The entire corpus finished in about a week on hardware we already owned.

The ensemble has one advantage over a frontier model besides cost. It is explicit about uncertainty. When its engines disagree, it leaves the character blank instead of guessing. A frontier model often does the opposite. It produces a perfectly plausible modern character, and unless you compare it with the original page you may never realize the historical form has disappeared. A blank is an error you can revisit. Silent modernization is much harder to detect.

What this adds up to

For anyone working with pre-1933 Korean print, the practical picture has changed. Current vision-language models can do more than transcribe these pages. The strongest of them can preserve historical orthography that is absent from most digital editions. It can also help one quickly digitize small (or large) amount of works. However for very large datasets cost can quickly skyrocket. It might be worth distilling some of this knowledge into a smaller, more specific model to save on API costs.

That does not mean OCR has been fully solved. The results are model-dependent, page-dependent, and, as far as I can tell, language-dependent. But for this particular corner of the problem, the gap between what was possible a few years ago and what is possible today is surprisingly large.

This is an extension of part of the presentation on using LLMs in research pipelines I gave for Machine Collaborators, series of talks organized by Charles Crabtree (Monash University) where researchers walk through how AI is being used in their work.

A recent product from OpenRouter called Fusion has multiple "cheap" models answer the same question, which they say gets within 1 point of the best "frontier" model at about half the cost. This isn't a novel idea, but the marketing is new. Pitting several models against one another has become a standard way to use LLMs. So what use might it have for empirical research?

Subscribe now

I’ve used a similar setup in two of my studies. In one, six models read about 4,000 open-ended answers to a survey question and sorted them into pre-defined codes, helping me to understand what people were thinking when they answered a survey experiment. In another, four models read 67 history textbooks published in South Korea and (using a technique that I developed) identified the set of words that best described how the textbooks related to national identity. The research questions and coding schemes are different, but the empirical instrument is the same. In both cases, I used a panel of open-weight language models, treated as independent coders, to vote.

Some thoughts on how this works, why I run mostly open models, how it relates to the “councils” being explored by Fusion and others, and what this does and doesn’t do for research. It works really well, actually. In terms of coding tasks it is on par with spending human hours on it. Its value, however, is very much about scale rather than certainty.

The method: open models as a panel of coders

Given a task where one may traditionally hire a trained research assistant (e.g. reading a piece of text and labeling what it is about), provide the same codebook and instructions to multiple models. Each model reads the text, is given the same instructions and codebook, and provides a single response (i.e. the label). The majority label is the classification, and the number of models that agreed on it is recorded as a measure of agreement.

Two features of this design are worth noting before I move on to describe the steps taken. First, the models are treated as coders. As such, they are tasked with assigning a label in much the same way a trained research assistant might. This stands in contrast to an approach that might be interested in the probability distributions provided by the models or any other measure of confidence (more on that below). Simply put, I ask the models to make a decision and to provide that decision “labeled” in a way that I can compare them to one another. Just as human coders can be compared by calculating (chance-corrected) inter-coder reliability (e.g., Cohen’s and Fleiss’ kappa), so too can the LLMs. Second, the voting is conducted independently. Not only are the models run at “temperature” zero (i.e., they are not permitted any “creativity”), they are also not permitted to view one another’s answers or otherwise deliberate. The models are still played off against one another, just at a later stage, when I line up the votes and look at where they part ways.

I run most panels on open-weight models because proprietary models behind API calls can change over time, and can't give repeatable answers if run elsewhere. Open weights models can be run locally with a fixed temperature and seed, and provide the same answers when run again in the same environment, making them suitable for collecting reproducible measurements. The model panel for the classification study includes one proprietary model, GPT-4o-mini, because it's a very cheap, fast, and capable reference coder that also has good performance on Korean and Traditional Chinese, which are relevant for those projects. The panel for the term extraction below is fully open weights, however, and GPT-4o-mini is only one voice in a larger open panel.1

An example of the method. One open-text response from a survey is read by six models that vote independently. The majority becomes the code, and the size of the majority is recorded as a reliability signal.

Three ways to convene a council

There are a few ways to use models in concert, and they aren't all after the same thing.

Fusion is, I think, the production version of this. The way it works is that a set of models answer in parallel, then a judge model reviews the outputs and produces a structured summary of consensus, disagreements, and missed items, then a calling model uses that summary to produce the final answer. It’s designed to optimize for answer quality at a given cost, but is not intended as a drop-in replacement. It is intended for cases where asking for multiple perspectives is valuable. The models often do come to different conclusions, but these differences are resolved in a way that is not exposed to the user.

Sakana AI’s Fugu uses a similar approach, with a trained orchestration model that selectively delegates work to its pool of agent models, verifying and synthesizing their results into the final answer. All this multi-model coordination happens under the hood, exposed to you as a single model API.

There are also two other related projects that I found from the computer and political science angle. First, there is Andrej Karpathy's llm-council, where multiple language models answer a question, then rate each other's answers while hiding the identity of the model, and then the "chairman" model answers the question using the ranked answers. And then there is Andrew Hall's llm-council-governance, which studies how the "decision rule" affects the outcome. He tries out 7 different governance rules, such as majority voting or deliberate then vote, using problems with known answers. The best performing governance structure was deliberate then vote, with an accuracy of 80.9%, compared to 71.7% for the strongest individual language model. There is also the more "academic" Language Model Council, where 20 language models write tests, answer them, and grade each other's answers. The authors show that the models can rank each other in a way that is more closely aligned with human rankings than any individual model.

Three ways to wire a panel of models. All draw on the same lineage. They differ in what they do with the disagreement.

These are similar enough that the models involved in the respective lineups overlap. Hall asks a council of Qwen, Llama, Gemma, and Mistral, and the panel that classifies below includes Qwen, Llama, and Gemma, along with three other models. But where this differs is that for Fusion and the councils, the ultimate goal is to distill the panel’s answers down to a single one, and in Hall’s case the best approach (deliberate then vote) is the one that produces the best single answer. In contrast, here I want to retain the spread of answers, because the spread is what I’m measuring. I view each model as a coder who is assigning a label to an observation, and I want to know if the coders agree or disagree, and how uncertain each coder might be.

Application one: classification

I will begin with classification. It is tied to a working paper I am writing titled “What Were They Thinking?” where I seek to find a way to use open-ended text (usually collected only as a manipulation check) to validate the constructs in a conjoint experiment. A conjoint tells you what respondents chose, but rarely whether they reasoned about it the way the design assumes, so I added the open-ended prompt “Why did you choose this person?” after the conjoint task. What I have are about 4,000 short answers from conjoint experiments on naturalization priorities. I have two corpora, one in Korean (South Korea) and the other in Traditional Chinese (Taiwan). My codebook contains 5 codes, including a “none of the above” code, and the codes focus on civic commitment, functional integration, economic focus, and identity concerns.

I code every answer with 6 LLMs: GPT-4o-mini, and five local Ollama models (Qwen 2.5 3B, 32B, 72B, Llama 3.1 8B, Gemma 3 12B). I assign a classification when at least four of the LLMs agree on that classification. It gives a Fleiss' kappa of ~0.74, which in content analysis terms is "substantial" agreement. Moreover, when more of the LLMs agree on a classification, each LLM is more confident in that classification - so when all six LLMs agree on a classification, it's probably safe to trust that classification. I deal separately with cases where the LLMs split 3-3 (either by human coding or some other way).

As more of the six models agree on a code, each model is also individually more confident in it. Most answers sit at high agreement. The split cases are the ones worth reviewing manually.

Two things make this more than a cost-saving trick. First, scale. Coding 4,000 answers to publishable standards, in multiple languages, with two trained coders and an adjudication protocol, would be a major project, and the next survey would be another one. It took the panel of models an afternoon to code the entire corpus. It could be done again, if set up to be replicable, and if the codebook is revised, with no trouble at all. It scales.

Second is quality. Rather than thinking of the models as a cheap way to essentially replicate the work of a human coder, think of them as a fast way to have someone reason through a particular answer to a particular question, given a particular context, in this case, a relatively short one. I did look at the ones where there was actual ground truth in the form of human coders and the panel disagreed, and usually I thought the label the model produced was the more appropriate one.2 Others have found similar results: check out the work by Ryan Briggs and others where they use language models to extract a lot of validated data from ~100,000 political science articles.3

I do want to note that not every disagreement is the model's mistake. Some show where the codebook could be clearer, and those are the useful ones to look at. You're ultimately tuning the codebook, and if your goal is accuracy, you should make sure the panel roughly matches whatever ground truth you have (i.e., human codes). When I first ran this, I got fair agreement with the human codes; looking through a batch of those, I saw that the models were following the codebook definitions for two categories, but those definitions didn't match what I had in mind (one too specific, a residual category too general). I updated the codebook definitions and re-ran the panel a couple times, and I was able to get agreement between the panel and the human codes from fair (kappa = 0.40) to substantial (kappa = 0.67), which is a bigger improvement than I got from switching models. Remember, the model is the coder, so you're essentially calibrating your coder with the codebook against however many human codes you have, and that is where most of the accuracy comes from.

Application two: finding the words that matter

Second, I will discuss a slightly different approach to using multiple LLMs. In Constructing the Nation (with Aron van de Pol) we examine how the representations of national identity in South Korean history textbooks have changed over the postwar period. However, in this project, there is no codebook that we apply. This is because the categories that we want to code for are the quantity of interest. Thus, we need to identify the words that represent national identity in the corpus, and then how the use of these words has changed over time.

If someone wanted to extract terms, or concepts, or even just words from a corpus, the standard method would be to use a topic model. And the standard topic model to use would be Latent Dirichlet Allocation. LDA, though, treats each document as a bag of words. It works by grouping words that appear together in documents. It’s very fast, and it doesn’t need supervised data, but it doesn’t take context into account at all. That can be a problem if the meaning of a word (especially if it’s a word like minjok, meaning something like nation or ethnos) depends a lot on the context it appears in, which is very often the case in the long, discursive chapters of the textbooks I’m working with.

Another approach using LLMs that I tried consisted of reading long parts of the corpus with 4 diverse LLMs and then asking them to propose terms associated with identity. If at least 3 of them proposed a term, then this term was added to a set of terms, if and only if it passed a frequency and semantic similarity test on the corpus. The 4 LLMs chosen were from 4 different countries in order to maximize their diversity. They were EXAONE (Korea), Aya Expanse (Canada), Qwen (China), and Gemma (USA). This method found 9 terms that defined identity.

Of the nine terms the four-model consensus defines, an embedding-based topic model identifies all nine, while classic LDA identifies five. Context-aware methods find terms that bag-of-words co-occurrence misses.

To check, we ran BERTopic, a topic modeling technique based on embeddings, and which, unlike a bag of words approach, does read the context of words. BERTopic also found all 9 terms, and with far fewer edge cases. In fairness, BERTopic does not read the whole textbook: the embedding model it uses only reads short passages at a time, because it can only handle a limited number of tokens at once.

An LLM hits the same limit, just a bigger one. If you were to copy an entire book into the LLM it would either reject the prompt or cut it down because it would be too long. But its window is wider, so it can read more context around a term. It can also read documents in large batches with overlap to cover the entire document.

As a side note, the ‘Classic’ LDA only finds 5 terms. The panel of LLMs does what topic modeling should be doing, finding the ‘important’ terms in a corpus. They do this in the context of the passage, whereas classic topic modeling only looks at word occurrence. For a corpus of long documents, this makes a big difference.

What LLM voting does and does not measure

A few caveats: this is just measuring inter-model agreement, not validity, and this is not a well-calibrated uncertainty estimate. The models are not independent (they are likely trained on a lot of the same data), so the guarantees of the Condorcet jury theorem don’t apply. If anything, I’d interpret this as a measure of inter-coder consistency between non-independent coders. (So, I’d expect 9 models to be less informative than 9 humans, for instance.)4

That’s worth remarking on, but I don’t think it’s the main thing here. Relative to the one set of human-adjudicated codes I had (a separate corpus, about entrepreneurs in North Korea, not the above), the models agreed with the humans quite a bit (not as much as the humans agreed with each other, but still a lot; and in the cases where the models and the humans disagreed, I usually sided with the model). The limit does impose a decent bit of discipline, though. If you can, I’d recommend including some human-coded ground truth, even if just for a few hundred cases, in any given project. If you want within-model confidence, you should probably derive that from the model’s token probabilities (i.e. its predictive distribution) rather than from the agreement count or the model’s self-reported confidence.

This last point is important, because if the language model is used as a substitute for a coder, its uncertainty should be handled like a coder’s. Even if the codebook is fixed and the temperature is zero, the language model will be more or less certain about its classification choices. This shows up in the log probability of the classification that the model chose, based on its predictive distribution for that classification. It is not the same as having the model also answer whether it is “HIGH” or “LOW” confidence in its classification choice, because that is simply another generated answer.

I am using log prob simply as a measure of uncertainty. It tells me how much probability the model assigned to the label it chose, according to its predictive distribution. The responses that are labeled, but with a low log prob, are what I am interested in looking at more closely. This is something that the panel does not give me. The log prob is a measure of how certain each of the models is, and the panel agreement is a measure of whether they are all making the same decision.

Is it worth it

It is, if the claim stays within the evidence. Here, I explored using a panel of open-weight models for research. Across two applications, it looks like this is a very useful tool. First, I show that a panel of open-weight models can classify texts just slightly worse than the agreement of human coders. This means that a panel can plausibly be used as a replacement for large numbers of human coders. Second, I show that a panel can also discover terms in a corpus; it outperforms standard bag-of-words topic models by leveraging contextual information to identify meaningful, defining terms. Taken together, these results illustrate the benefits of using open-weight, diverse panels of LLMs to ensure replicability and the value of analyzing disagreement.

A panel of LLMs can be better than a single LLM. However, when the LLMs agree, that doesn't mean they are right; it means the particular LLMs in the panel agree. The two only line up if the LLMs are both accurate and diverse. Despite this, I think it is worth building a panel and paying attention to where it disagrees. (Note that this approach still requires lots of human-in-the-loop, both pre and post LLM run, to refine the codebook and adjudicate cases.)

Using Claude Code

If you want to do something like this, the most relevant of my open-source Claude Code skills (codifications of steps I follow when using agents, to improve how reliably I can get them to do things) are model-council-voting (for running a panel and seeing how they agree), text-classification (for designing and validating a codebook), and llm-calibration-logprobs (for getting the model’s confidence in each of its decisions, in terms of log probabilities of the tokens it emitted in making that decision).

Subscribe now