Most of my work with a command-line agent is writing prompts. Claude Code and Codex do best when I can provide as much information as possible. They excel at determining what I want from the latent constructs in the text I provide. Detailed prompts of this kind easily run three hundred words long (and longer), and typing three hundred words for every prompt, all day, is a kind of energy-sapping drudgery. The rational response is to shorten the prompt, which also degrades the answer and the work being done.

Subscribe now

Earlier this year, I discovered just how good the speech-to-text technology is and created setups for my desktop machine which runs on Linux and my Macbook Pro. I stopped typing most of my prompts and started dictating them. I tap a hotkey and talk at the terminal through my mic for as long as the thought takes. Another tap ends the recording, and about a second later the cleaned-up text is sitting at my cursor. Accordingly, my prompts got longer and the answers and outputs got better, because a language model is very good at pulling intent out of even meandering speech. It needs volume and specifics, and speaking is the cheapest way to supply both. I still type anything that must be exact, such as slash commands, file paths, flag names, and model IDs, but almost everything else I say out loud.

This configuration has been especially useful for work that benefits from repeated explanation and close supervision. I use it to iterate on my own writing, draft emails, develop prose from rough ideas, and produce detailed feedback on student assignments, particularly writing assignments. In that setting, the limiting factor is often not what I have to say, but how much of it I am willing to type. Thorough feedback requires explanation: identifying the problem, showing why it matters, suggesting a revision, and sometimes distinguishing between several possible ways forward.

The advantage is not simply that speaking is faster. It also makes it easier to think through a response in full. I can talk through an argument, qualify a judgment, add examples, and revise my own assessment as I go. I then clean it up through edits, mostly typed but also some spoken. For student advising, this has been a big improvement. I can give more specific and more useful feedback. I have not subjected this to an A/B test, but my impression, and I am open about this with my students, is that the feedback is much better than what they are used to receiving.

Typing, however, is not obsolete. It remains better for anything that must be exact. The most effective workflow combines the two. As I will explain below, there is some modification of your voice as it is produced as text, especially if you want it to be clean. This post documents the improvement in the underlying technology, then the setup I run on Linux and macOS. Both configurations are public, hyperwhspr for Linux and macwhspr for the Mac. Speech-to-text is useful for all sorts of cases, but my reflections from here on are based primarily on my experiences from interacting with Agentic AI.

Speech recognition is good now

For a long time, dictation software produced enough errors that correcting the transcript took longer than typing it would have. In 2011, Microsoft researchers cut the word error rate on conversational telephone speech from 27.4 percent to 18.5 percent by swapping Gaussian mixture models for deep networks (Seide, Li, and Yu 2011). A system that still got nearly one word in five wrong counted as the breakthrough of its day. By 2016 the same benchmark was at 5.8 percent, alongside a measured 5.9 percent for professional human transcribers (Xiong et al. 2016). A year later it was 5.1 percent (Xiong et al. 2017).

OpenAI’s Whisper made high-quality speech recognition widely accessible. Trained on 680,000 hours of audio, it transcribed LibriSpeech at 2.7 percent error zero-shot and held up on accents and noise where earlier models collapsed (Radford et al. 2022). In 2023 it went behind an API at $0.006 USD per minute (OpenAI 2023), which is 36 cents per hour of speech. The 2025 generation, gpt-4o-transcribe, beats Whisper across the FLEURS language benchmark by OpenAI's own evaluation (OpenAI 2025). These figures come from different test sets. They are not directly comparable, but together they illustrate the broader improvement in accuracy, quality, and cost.

Once transcription accuracy became good enough, the remaining irritation was the pause after speaking. The 2026 releases address this, with streaming models that transcribe while you talk instead of after you finish. ElevenLabs claims under 150 milliseconds for Scribe v2 Realtime (ElevenLabs 2025), and OpenAI's gpt-realtime-whisper streams over a WebSocket at $0.017 per minute (OpenAI docs). I switched both of my machines to it this week, with notable improvements.

The setup

The pipeline is the same on both machines. A hotkey starts recording. Audio streams to the transcription model while I speak. A second tap commits the buffer, and the final transcript comes back in about a second. A small LLM then repairs the transcript, and the result is pasted at my cursor in whatever window has focus, which is usually a terminal and occasionally a draft like this one.

hyperwhspr, for Linux

On Linux the recorder is hyprwhspr, and scdenney/hyperwhspr is the reproducible configuration pattern I run under Omarchy/Hyprland. It holds the streaming backend config, the cleanup hook with its constrained prompt, the vocabulary file, the systemd service with its Wayland fixes, and the calibration command that uses the cleanup log to make new vocabulary rules. The README keeps a dated known-issues table, including this week's switch to the streaming model and other notes added by AI agents or by me.

macwhspr, for the Mac

There is no hyprwhspr on macOS, so scdenney/macwhspr is the whole setup in one small repo. An idempotent installer copies everything into place. A small Python daemon records audio with SoX and streams it to the transcription service. Pressing and releasing the Globe key (my hotkey of choice) starts or stops recording, with Karabiner and Hammerspoon handling the keyboard shortcut and a small on-screen status indicator. launchd keeps the daemon running in the background, and if a transcript ends up in the wrong window, Ctrl-Cmd-V opens a list of the last twenty recordings so it can be recovered and then pasted where you want. The repository contains the complete setup. Aside from the platform-specific content, the transcription, cleanup, calibration, and logging pipeline is the same as on Linux.

The cleanup stage is an underrated part of this setup that I did not quite appreciate until setting it all up. The transcription model returns raw text with filler words, false starts, and little punctuation. A tightly constrained GPT-4.1 mini prompt restores punctuation, paragraph breaks, and consistent formatting, but it is explicitly told not to change the meaning or answering the content. If I dictate a question, I get back a well-formatted question, not an answer. Every raw and cleaned transcript pair is logged, and a calibration command uses those logs to update a vocabulary file with preferred spellings, names, romanizations, and other recurring corrections — this is how you can customize your use. Over time, the system gradually adapts to the way I speak.

Below is a recording loop example from my Linux machine. Hotkeys start the recording, the pill streams the live transcript while I speak, and a second pushing of the hotkeys send the cleaned-up text to the Claude Code prompt.

Latency differences

Until this week both machines uploaded the finished recording after I stopped talking, which meant the wait grew with the length of the dictation. The streaming backend does the work during the speech itself.

I measured both backends on identical audio, synthetic speech streamed at real-time pace over the same network. On a 6.5 second clip, batch took 1.7 seconds after stop and streaming took 0.9. On 94 seconds of audio, batch took 5.9 seconds and streaming took 0.9 again. Across the lengths I tested, the streaming wait was effectively flat at about a second. My most recent log has a 128 second dictation, a spoken complaint about a LaTeX table layout in a manuscript — a commonly written and voiced / yelled complaint, I am sure. The transcription wait alone was five seconds. Under the streaming backend, a dictation of that length now finishes in about one second. For a workflow built on dozens of dictations a day, that dead time adds up. The newer model is much better, in other words.

Each cell in that chart is a single run, though the production logs agree with the pattern. Streaming costs $0.017 USD per minute against $0.006 for batch, so a two-minute dictation costs about three cents. The realtime model accepts no vocabulary prompt, so domain terms ride on the cleanup stage instead. A WebSocket connection has more failure modes than a file upload, and both repos keep the batch path one config line away. The last caveat is that the audio leaves the machine. If that is unacceptable, hyprwhspr supports a local ONNX backend, at the cost of about 5.7 GB of VRAM and maintaining the model yourself. I found the local option usable, but not good enough to justify those additional demands.

Subscribe now

What a month of this costs

The question of cost comes up whenever I describe this setup, so before writing this post I pulled thirty days of logs from both machines. Between June 13 and July 13 I dictated 662 times across the two machines, which comes to 7.7 hours of speech and 54,796 words. The transcription bill for the month was $2.77 USD, and the cleanup model added another $0.27, so the total was $3.04, or about five and a half cents per thousand words. The whole window ran on the older batch backend, and at streaming prices (my latest setup) it would come to about $8.11. For me, the difference is negligible. The deciding factor is the latency.

Make the switch

Typing still has an imporant role, just to be clear. It is better for writing commands, file paths, quotations, citations, identifiers, and anything else that must be exact. It also remains an important part of writing itself. I am not about to dictate an entire scientific manuscript. Drafting, revising, and polishing still belong at the keyboard — at least for me, for now.

But typing should not be the only way you work with AI agents. Restricting yourself to the keyboard either limits the context you provide or makes providing that context unnecessarily slow and tiring. The better workflow is hybrid. Explain the problem out loud, let the transcription and cleanup stages produce a first draft, then edit it if needed, adding the precise commands, paths, or other details that only typing can reliably supply.

Speech recognition is accurate, inexpensive, and nearly immediate. Talking to an agent is no longer a novelty or an accessibility matter. It is a practical way to give these systems that are coming to dominate our computational and other kinds of pipelines the detailed instructions they require. If you do substantial work with agents and still type every prompt from beginning to end, you are imposing a bottleneck the technology has largely removed.

Note: Data and replication code for the analysis provided here are available on GitHub.

The United States turns 250 this year, and the old argument over what it means to be American remains as intense as ever. What better moment than now to engage in that age-old debate of what it means to be an American? Flatten by media coverage, the debate usually gets cast as two options. Either the country is a creed, a set of ideas anyone can sign up to, or it is a heritage, as a matter of belonging that you inherit by birth and/or blood.

That two-sided framing is not merely an artifact of punditry. It dates back at least to the historian Hans Kohn, who, in 1944, distinguished between civic and ethnic nationalism. Civic nationalism defines the nation in political terms, as a community of citizens bound by shared institutions and ideals. Ethnic nationalism defines it in cultural and genealogical terms, as a people united by common ancestry, language, or heritage. At its core, Kohn's distinction concerned how people answered a fundamental question: who belongs to the “circle of we"? To a large degree, this civic-ethnic distinction has organized the study of nationalism ever since. Scholars have, of course, challenged how clean the divide really is. As Anthony Smith argued, even civic nations draw on ethnic myths and symbols.1

Subscribe now

As public opinion surveys became a cheaper and more reliable way to measure public opinion, political scientists and sociologists began asking people directly who belongs to the nation and how they relate to it, rather than inferring those views from elite rhetoric. Deborah Schildkraut’s Americanism in the Twenty-First Century and Elizabeth Theiss-Morse’s Who Counts as an American? both show that Americans’ views are more varied than a simple creed-or-heritage divide suggests.2

When Americans are asked directly, they do not divide neatly into two camps. Using a method that identifies recurring patterns across attitudes toward national belonging and national pride, Bart Bonikowski and Paul DiMaggio distinguish four dispositions. Their typology, however, was developed from a 2004 survey and only later compared with data from 1996 and 2012. The question is whether those same four types still structure American national identity today.

I replicated their approach using the International Social Survey Programme’s (ISSP) National Identity module, which asked the same battery of questions in 1995, 2003, 2013, and 2023. I find, again, the same four identiy types. Americans do not sort into two competing visions of the nation, but four. What has changed over the past three decades is the relative size of those groups. One has remained remarkably stable, while the others have risen and fallen.

Two questions, four Americas

The types follow from two basic categories of questions. First, who counts as truly American? Is it about being born here and being Christian, or about citizenship and respecting the country's institutions? Second, how warm you feel about America. How proud are you of the country? Crossing these two dimensions gives you four identity types.

The horizontal axis captures who counts as truly American, from inclusive (citizenship and shared institutions) to exclusive (birth and religion). The vertical axis captures national pride. The types and their attitudes are then as follows:

• Ardent: “hot” nationalists, scoring high on everything. They want high boundaries for belonging and are proud across the board.

• Restrictive: people who also draw distinctive boundaries of belonging and are sure the country is superior to others, but their pride in its actual achievements runs cooler than ardent nationalists.

• Creedal: those who are proud and inclusive. They anchor belonging in citizenship and institutions rather than birth or ethnicity/religion, and they are less likely than ardent or restrictive nationalists to say the country is better than others.

• Disengaged: those “cool” on all of it, rejecting or downplaying any sort of national identification.

You can see each type's profile in the survey responses. The clearest divide is not national pride, which most engaged Americans share, but the boundaries of belonging. The ardent and restrictive types are much more likely to say that being Christian is part of being truly American and that the United States is superior to other countries. Creedal Americans reject both of these items as conditions for national membership. They are low on the ascriptive criteria (born here, be Christian) and on superiority, but high on the civic criteria (citizenship, respecting institutions) and moderately proud, the signature of an inclusive-but-attached patriotism. The data presented so far already complicates the creedal-versus-heritage framing.

How the types are measured

Since this Substack is meant to be somewhat technical, a word on how the types are constructed. The data come from the ISSP National Identity module, which fielded comparable nationally representative U.S. surveys in 1995, 2003, 2013, and 2023. I use the sixteen items included in all four waves.

As the figure above shows, the questions fall into three broad domains: who counts as truly American, how proud respondents are of different aspects of the country, and whether they express a sense of national superiority or unconditional loyalty. The first domain distinguishes civic criteria, such as citizenship and respect for institutions, from ascriptive ones, such as birthplace and Christianity. The first eleven items use four-point response scales; the remaining five use five-point agree-disagree scales.

The identity types are derived from latent class analysis. The method treats the population as a mixture of unobserved groups, each defined by a distinct pattern of responses. It estimates both the size of each group and the probability that each respondent belongs to it. Unlike factor analysis, a commonly used alternative in the study of national identity, which places respondents on one or more continuous dimensions, latent class analysis identifies qualitatively distinct types of people whose responses cluster together. That makes it well suited to studying national identity when the question is not how much nationalism people express, but how they combine different ideas about belonging and national pride. Bonikowski and DiMaggio adopted this approach, which is why "creedal" here refers to a type of respondent rather than a high score on, say, a creedal scale.

I estimate latent class models with one to six classes and selected the four-class solution based on model fit and interpretability. The classes are well separated, and the same four-type structure appears in all four survey waves.3

What changed

The next question is how these four conceptions of the nation have changed over time. Below is the proportions belonging to one of the four over the last thirty years. What do we see?

Two things stand out. First, the two hot types have receded. The September 11 terrorist attacks produced a sharp but temporary surge in more exclusionary forms of nationalism. The ardent type reached its peak in 2003 before steadily declining, while the restrictive type has fallen below its pre-9/11 level. Second, the creedal type experienced the inverse of that surge. Its share fell sharply in 2003 but returned to its long-run level of roughly one-third of Americans by 2013, where it has remained since. Across three of the four survey waves, the inclusive conception of the nation is the modal form of American nationalism.

Perhaps most striking is the growth of the disengaged. In 1995, they accounted for about one in eight Americans. By 2023, they had become the largest of the four groups, comprising more than one-third of the population. Because these are repeated cross-sectional surveys rather than panel data, we cannot conclude that individuals moved from the ardent or restrictive types into disengagement. But the aggregate pattern is consistent with exactly that possibility.

This finding should hold our attention. In 2023, the most common way of relating to the nation is detachment. It is hard not to make this connection to a broader retreat from shared life and experiences. Interpersonal trust has fallen, while community and religious membership has thinned out. Isolation has risen to the point that the Surgeon General has called it an epidemic. The problem is less that Americans hate one another's idea of the country than that a growing number are opting out of having one at all.

Politically, the next obvious question is whether this shift has unfolded similarly between left/right partisans. Looking separately at Democrats and Republicans show both encouraging similarities and notable differences.

The most striking finding is the stability of the creedal type across the partisan divide. Throughout the past three decades, roughly one-third of both Democrats and Republicans have held an inclusive but proud conception of the nation.

Where the parties differ is in the trajectory of the other three types. Among Democrats, the hot and boundary-drawing types declined, with most of that shift moving toward disengagement, which now accounts for more than four in ten Democrats.

Among Republicans, the restrictive type remained relatively stable, although it now appears to be declining as well. Instead, the defining change over the past three decades was the sharp post-9/11 surge in ardent nationalism, followed by its steady decline. Disengaged Republicans also became more common, although they remain much less prevalent than among Democrats.

The case for liberal nationalism

The creedal conception of the nation has proven more durable than the others. For those who see liberal nationalism as the most desirable basis for democratic politics, that is encouraging news. Creedal Americans define national belonging in terms of citizenship and shared institutions rather than ancestry, while expressing pride in their country without believing it is superior to others. This is what political theorists call liberal nationalism. It is, among self-identifying nationalists, a normatively desirable form of thinking about community in the nation-state era.

The central claim of liberal nationalism is straightforward. A shared civic identity can sustain liberal institutions by encouraging strangers to see one another as members of the same political community. Rather than defining the nation by ethnicity or religion, it defines membership through citizenship and shared democratic institutions.

The distinction is not merely theoretical. It also lies at the heart of one of the country’s most consequential constitutional debates: birthright citizenship. Creedal Americans place relatively little weight on being born in the United States as a condition of being “truly American,” but they overwhelmingly regard citizenship as essential. The logic of the creedal conception therefore points toward jus soli — citizenship, rather than ancestry, is the basis of national belonging. The Supreme Court’s recent decision In Trump v. Barbara (2026), reaffirming birthright citizenship under the Fourteenth Amendment, reflects this civic understanding of national membership.4

In recent years, a growing number of scholars have argued that liberals should reclaim this inclusive form of nationalism rather than cede the idea of the nation to more exclusionary visions. Writers making that case for America, from Noah Smith to Alexander Kustov and others, argue that the strongest foundation for national identity is one rooted in citizenship rather than ancestry, open to newcomers, and confident enough in its ideals to welcome them as fellow citizens. Attempts to bypass national identity altogether, whether through cosmopolitanism or other forms of supra- or non-national identification, they argue, simply leave the language of nationhood to the demagogues.

My findings suggest that this civic conception of the nation is more than a normative ideal. While ardent, restrictive, and disengaged forms of national identity have expanded and contracted over the past three decades, the creedal type has remained remarkably stable, accounting for roughly one-third of Americans in every survey. Nor is that durability merely an American curiosity. In my comparative study of twenty-nine countries, the United States emerges as the clearest example of civic nationalism: the country where commitment to democratic rights most strongly constrains hostility toward outsiders.5

Subscribe now

The road ahead

So the mood on America's 250th should be one of cautious optimism. The meaning of the American nation remains unsettled. The “bets still on the table”. The civic conception of the country has endured, exclusionary nationalism has not become dominant, but a growing share of Americans remains disengaged rather than committed to either vision.

There is, to be sure, a real loss of attachment. But that also leaves space for persuasion. As Ross Douthat argues, America’s enduring strength is its capacity to continually renew itself. This is a contested nation, but diversity is somethig most Americans think makes the country stronger, anyway.

The central challenge, then, is not simply to contain exclusionary nationalism, which is always going to exist in some form or another. It is to build a more compelling alternative. The evidence suggests that the most durable conception of the American nation is also the most small-l liberal. It is the one that grounds belonging in citizenship, shared institutions, and common political commitments rather than ascriptive characteristics. For advocates of liberal nationalism, the task is therefore not only to oppose exclusion, but to persuade the growing number of disengaged Americans, especially on the left, that the American experiment is still a project worth identifying with and investing in.

Earlier this month, a widely shared thread on X pointed out that a prominent political scientist had a half-dozen fabricated citations in a recent article on South Korean politics.¹

The reaction was about what you would expect: disappointment, mostly, and, in some quarters, shock. To be sure, no instructor would accept this in a student paper, and fabricated references in a published article must be addressed, whether through a correction or a retraction. Given the number of references that appear to have been fabricated, I suspect that most people following the case expect the latter.

Subscribe now

Still, the speed and intensity of the response are worth pausing over. A case like this deserves more care than it has received so far.

First, we do not actually know how those citations got there. It is unlikely but entirely possible that the fault lies with the journal rather than the author. Many of us have concluded that this particular outlet does not meet the standards it advertises, and here it was turning an article around quickly to address fast-moving events in Korea. Under that kind of pressure, it is not far-fetched that an editor ran the manuscript through an AI tool to tidy or reformat the references and that the tool quietly rewrote them into works that do not exist. I do not know that this happened, and at this point I would have expected this information to have been brought to light, but neither does anyone else.

Calling out errors of this kind is entirely fair. But a public accusation of malpractice directed at an individual had better be well founded, because once it is made, it cannot easily be undone. Reputational damage travels quickly and lands hard; any later correction, if one comes at all, rarely receives comparable attention.

This moment also calls for a dose of humility. The ordinary, almost boring parts of academic work, such as locating a source and formatting a reference, have been disrupted. The tools responsible are exceptionally good at making users feel that they have completed the work correctly when, in fact, they have not. Even otherwise careful people can be duped by their personal robot.

Much of the anger on display, I suspect, reflects a broader anxiety about what AI is doing to the academy. Before making an example of someone, particularly someone with a record of excellent work, we should be far more certain about what actually happened. We should also place the case within the wider structural conditions that made it possible.

To be clear, I do not know how this happened. The most plausible in my mind is not calculated fraud but misplaced confidence in a technology that routinely produces convincing falsehoods. That does not excuse the error. It does, however, change what we should learn from it. The structural failure may be less an individual lapse than a culture that has adopted powerful tools without establishing equally powerful norms of verification. The good news is that this is a problem we can fix — and I have some solutions to try out. But first, we need to understand how we got here, and why the machine is so good at deceiving us.

Old habits die hard

Type a scholar’s name and a couple of keywords into Google Scholar, and you get a list of citations. Use a few of them without reading past the abstract because you already know roughly what that person argues, and you have done one of the more mundane tasks of writing. AI is just exposing an inconvenient truth. We do not (closely) read every source we cite. Some citations are indeed used as evidence, and some are presented as a kind of performance. They signal that we know the literature or that we have cited whoever a reviewer expected or explicitly asked us to. A chatbot like ChatGPT or Claude, in these early days, feels like the same exercise. You ask for the literature on a subject or particular claim, and it gives you names you recognize along with titles that look right. The problem is that some of those titles do not exist.

None of this is new. Take one prominent example. A decade ago, before anyone could blame a model, a prize-winning history of North Korea was found to have cited nonexistent or irrelevant sources in at least sixty-one places. The author returned his award and left his chair.2 The much less catastrophic version of the problem is ubiquitous. It includes use of the wrong DOI, a page range off by two, or a citation to a real paper for a claim it does not actually make. Hallucinations in all but name. What AI adds is ease of use and an appearance of accuracy, which contributes to the major structural issue we are all dealing with now.

Why the machine makes things up

The difference between Google Scholar and an AI chatbot is that the search engine (usually) returns sources that exist. A large language model returns things that are probable. It is neither a database nor a search engine. It generates the next most likely token given everything before it, so when you ask for a citation, it produces a string outcome that looks like one. The string has an author, a year, a title, a journal, and a volume because that shape is overwhelmingly probable wherever citations appear. Nothing in that process checks the reference against a master list. The model is indifferent to whether the work exists, a machine version of Harry Frankfurt’s “bullshit” — unconcerned with the truth of what it produces. A recent paper from OpenAI argues that the problem is more deeply embedded in these systems than we like to admit. Our training and evaluation procedures reward a confident guess over an admission of uncertainty, so models learn to answer confidently rather than abstain or admit ignorance.3

How often does this happen? In one controlled study, GPT-3.5 fabricated 55 percent of the citations it generated for literature reviews, and a newer model (GPT-4), while considerably better, still fabricated 18 percent.4 An audit of two and a half million biomedical papers found that the rate of fabricated references had risen more than twelvefold since 2023, to about 57 per 10,000 papers by early 2026. The steepest increase accompanied the arrival of new AI writing tools in mid-2024. Most of those references had already passed peer review, which is a lot to ask of reviewers who would have to chase down every source.5 The problem reaches well beyond the academy. In 2023, a federal judge fined two New York lawyers five thousand dollars after they filed a brief built on six judicial opinions ChatGPT had invented. The lawyer had asked the chatbot to confirm that one of them was real.

While we do not know how the fabricated citations in the case presented in the introduction of this post were produced, the likeliest account is the simplest one. A single prompt to a general-purpose chatbot, with no tools or documents to check against. That is the most fragile way to use this new technology, and some degree of hallucination in the response(s) is close to guaranteed. However, systems designed to retrieve and read real sources before answering do far better, even if imperfectly so. When Stanford tested the retrieval-based legal tools that vendors marketed as “hallucination-free,” they still returned false information in something like one answer in six. But this is at least a sign of things moving in the right direction.

At the very least, we need verification systems that can find hallucinations and other errors.

Responsible all the same

We are accountable for what we publish. There is no better safeguard than reading our sources carefully, checking every citation, and refusing to include anything we cannot verify. Fabricated sources are unacceptable whether they come from the poor use of an LLM, a tired research assistant, or our own carelessness in formatting.

For students, that standard has to be explicit, and at my own institution it is. While I am generally unimpressed with how universities — my own included — cast AI as primarily a tool for cheating, Leiden University’s Faculty of Humanities correctly defines fraud as any act or omission that prevents a correct assessment of a student’s knowledge. Its examples include the unauthorized use of AI software and the invention of research data.6 Its guidance on generative AI is more direct: “GenAI generates language, not information … If asked to produce references, many LLMs will simply invent non-existent citations.” A fabricated citation misrepresents the evidentiary basis of the work, which is exactly what assessment is meant to evaluate. Under a strict reading, it is indeed fraud, and the heaviest penalty on the books, exclusion from all examinations for a year, should be deterrent enough.

A field-wide response is going to be harder than a local one, and the distance between what these policies seek to resolve and what actually happens seems quite wide. The instinct here seems to be to police the problem out of existence. Rooted in a commitment to supporting good science, it is understandable. In mid-May, arXiv announced a one-year ban for authors caught submitting work with hallucinated references, with any later submission required to clear peer review elsewhere first.7 The motivation is right, but the mechanism is not. I think the verdict that the policy is “welcome but unenforceable” is the correct one.

The early evidence agrees. When one peer-review service ran a reference checker over two hundred papers posted to arXiv after the ban, more than one in four still carried a hallucinated citation. Despite threats of punishment, obviously people are using AI for research. The systems are too good, and too useful, for prohibition to hold. People will use them to write and to think, whatever the rules say. In all likelihood, messages meant to deter, to the extent that they are heard at all, are probably just driving an underreporting of AI use because of social desirability.

We need a different approach here. Teach people to use these tools well, in the open, and turn the same technology around on the problem it has exacerbated. Build verification into the places that matter, such as a journal’s submission desk, and help writers — senior authors and students alike — such that auditing a reference list, or checking whether a source actually supports the claim attached to it, becomes routine. Done properly, these tasks could be performed more systematically and reliably than they were before ChatGPT. This shifts the emphasis from telling people what not to do toward giving them something better to do.

Skills / chatbots

The first thing we need to do is stop treating chatbots as the default interface for serious research and writing. Instead, build (or use) skills for use with agentic AI.

A skill, in the agentic AI sense, is a structured set of instructions that constrains how an AI system does a task. It fixes the steps the model takes and the tools it may use. It also specifies how the model checks its own output. A one-shot prompt invites the model to be creative. A skill takes that freedom away. It can direct a model to dispatch agents and to look up sources in real indexes and distinguish what it actually confirmed from what it merely guessed. The fabrication problem is fundamentally a model answering from probability instead of from evidence, and a well-built skill helps close off that option.

I keep a small library of these for my own research, the Open Science Skills. Three of them are built for the problems we are entertaining here.

The first is a reference checker (citation-check). This most directly addresses the problem under consideration here. It inventories every in-text citation and every entry in the bibliography of a paper, then runs the checks on them. It confirms that each cited work exists, using programmatic indexes like Crossref and OpenAlex rather than the model’s memory. It confirms that a DOI resolves to the cited paper rather than merely to some paper. It cross-checks a suspicious title against the named author’s actual body of work. And it will not call anything fabricated until it has failed to find it by title and by author. What comes back sorts the serious problems from the cosmetic ones and flags the uncertain cases for a human. It is not perfect, but it works.

The second addresses a different, arguably much deeper problem. It is one that is much less likely to go viral on X. A citation can be real and perfectly formatted and still not support the claim it is associated with. The source-claim checker (fact-check) reads the cited source and asks whether it supports the specific sentence. It flags overclaiming and direction errors. Notably, it needs the sources to read. As I’ve written the skill, you cannot just direct it to the internet or ask it to “get creative”. This takes us to the third skill.

The knowledge-base scaffold (research-repo) builds a project around its source library. The original formats (e.g., .pdf) are kept in one folder while another holds the machine-readable conversions the checker reads (markdown, preferably). Each is keyed to a bibliography entry for a document you have actually filed so there is a direct connection between source and its bibliographic metadata. The claim checker refuses to run against an empty or half-built library. To confirm you have represented a source correctly, you have to actually have the source — or, where the source is not available, a file that faithfully summarizes it.

Try it yourself

None of this is as elaborate as it might sound, and the best way to figure this stuff out is to just do it yourself. I have put up a small, self-contained walk-through on a synthetic manuscript seeded with deliberate errors, so you can watch the checks fire away. It is the first demo in a teaching hub (working prototype) I am building for students and colleagues at Leiden University, with the setup guide and slides alongside it.

We are in an odd period. The technology is too useful to ignore, yet unreliable enough to embarrass or even discredit us. At the same time, we have not used it long enough to develop sound habits around it. That is a dangerous combination.

The answer is not a moratorium or a rejection of the technology. Nor should it be a hunt for the next person to make an example of on social media. We need humility about how easily these tools can fool us, some grace for those who have misused them, and stricter standards for how we use them. These positions are not contradictory. We can hold all three at once.

Subscribe now

Correction: It has since been clarified that the article published in Korea Observer was an "annual review" reflection and was therefore not subject to peer review, which explains its quick publication timeline.

OCR has changed substantially in the last few years. For clean, modern, printed English, the task no longer usually requires a dedicated OCR engine, a domain-specific training set, or extensive preprocessing. Current vision-language models can often transcribe such material directly from a prompt, sometimes zero-shot and sometimes with a single example. In that limited sense, printed Latin-script OCR for high-resource, modern material has become a largely infrastructural problem rather than a central research challenge.

Historical documents are still harder. Old paper, damaged scans, unusual layouts, handwriting, and older typefaces all push modern models well outside their comfort zone. But even here the progress has been striking. Platforms like Transkribus routinely read European manuscripts that would have seemed hopeless a decade ago, and projects such as Loghi report similarly strong results on historical print. OCR has become much more accessible.

Korean, and by extension CJK1 and other non-Western scripts, is a different case, and early-twentieth-century Korean is different again. Pre-1933, Korean was written in an orthography that has since fallen out of use. It spelled words closer to how they sounded than to a fixed stem, and it kept letters the modern keyboard has dropped, among them the dot vowel arae-a (ㆍ) and a set of old consonant clusters. The spelling we read today is the Unified Orthography of 1933 (한글 맞춤법 통일안), the morphophonemic standard carried by Chu Sigyŏng’s lineage in the Korean Language Society, and it became the convention only after a long argument that ran through the colonial period and resurfaced in the spelling crisis of the early 1950s.2

Modern Korean OCR remains less stable than the English and European-script cases above. On the Korean portion of the PM⁴Bench benchmark the leading system changes with each new model release, and in the current version Gemini 3 Pro ranks ahead of the newest open model, Qwen3-VL, which does not improve on its own predecessor for this task. This should be read as a relative ranking within a weak field rather than as evidence of reliable transcription. The absolute scores remain low.

In one of my projects, I need OCR labels for about ninety-four million individual character images. Because I study the appearance of historical type rather than the content of the text, it is important that older Korean characters are transcribed as they were printed, rather than silently modernized. This post explores how well current models actually do.

A few tests pages

The page is from Sonyŏn (소년), a children’s magazine published in 1908, and contains a chain-rhyme word game. It is printed in vertical columns read from right to left, combining (Old-)Korean Hangul and Hanja on the same page. Material like this has traditionally been difficult for OCR systems.

The part I cared about was not whether the models could produce readable Korean. (Though even this would be helpful for many historians.) It was whether they would OCR Korean letters as printed. Before the 1933 spelling reform, Korean used forms that later disappeared or changed in ordinary writing. These include the dot-vowel arae-a (ㆍ) and several old consonant clusters. A model can read a page fluently and still normalize those forms into modern Korean. For most readers the output still looks right. For my purposes, it is wrong.

The first page was a narrow test. It happened to contain two old cluster forms, ᄭᅡ and ᄯᅱ. A modern transcription would normally write these as 까 and 뛰. GPT-5.5 kept all six occurrences of those old clusters on every run. The previous GPT model had modernized all six. That was already a meaningful change, but it did not prove much beyond this particular page.

So I tried five more pages from Sonyŏn, using prose and dialogue rather than word games. These pages contained a wider range of old Korean forms: arae-a in forms such as ᄒᆞ and ᄂᆞᆫ, several ㅅ-cluster groups such as ㅺ, ㅼ, and ㅽ, and even the rarer triple cluster ㅴ. This was the real test. The question was no longer whether a model could preserve two repeated forms on one page, but whether it could keep a broader old-orthography repertoire across different pages.

The current models can preserve old Korean forms, but not with equal reliability. Preservation is still something to check, not something to assume. In practice I treated an old form as trustworthy only when different models produced it, because a single model can also hallucinate plausible old-looking Korean.

The dedicated OCR engines were not trained on any form of old-Korean and not surprising that they performed the poorest. Mistral OCR, DeepSeek-OCR, Baidu’s Unlimited-OCR, GLM-OCR, all read the page and then modernized the old letters or fell apart, one of them looping a few character lines a few hundred times, another reading the Hangul as Chinese and drifting into a gibberish. Qwen3-VL was different. It read the Korean fluently, but silently regularized every older form into modern Korean. If your goal is simply to recover the text, that is a respectable result. If you care about preserving historical orthography, it is not.

Going back further, to 1896

After Sonyŏn, I wanted to know how the models handled even earlier Korean print. As a final test, I tried the front page of the Tongnip Sinmun (독립신문, The Independent) from 1896, the first privately published Korean newspaper, set entirely in Hangul in the full old orthography with arae-a in almost every word. The only scan I could find is small, a single 960-pixel page, which makes it a hard test on top of an old one. For once there was something solid to check it against. The Korean Wikisource was manually transcribed, so this is the one page in the whole exercise I could measure against ground truth rather than judge by eye. It should also be noted that this newspaper image is denser and has a more difficult layout then the previous magazine had.

Most models struggled. Claude Opus 4.8 transcribed only about a sixth of the page, even reversing the masthead. Qwen3-VL reached roughly two thirds, but regularized the spelling throughout. Both ended up at around ninety percent character error against the Wikisource transcription.

Gemini 3.1 Pro was the clear exception. It read essentially the entire page, from the masthead through the advertisements and price table to the end of the editorial, about ninety-six percent of the page by length. Compared against the Wikisource transcription, it reached roughly seventeen percent character error while preserving 176 of the page’s 182 arae-a. On a 960-pixel scan of an 1896 newspaper, that amounts to a complete transcription of a full old-orthography page.

GPT-5.5 failed for a different reason. It did not misread the page so much as fail to read it at all. Tongnip Sinmun is one of the most famous newspapers in Korean history, and its first issue has been reproduced countless times. When shown the thirteenth issue, GPT-5.5 instead reproduced the first: its date, issue number, and opening lines, at both image resolutions. Against the actual page, the result is about eighty percent character error. It is a useful reminder that, for well-known historical documents, a model can sometimes retrieve what it already knows instead of reading what is actually in front of it.

Subscribe now

Why this matters to someone working with the sources

The interesting part is not the benchmark. It is what happens to the historical record.

When the National Institute of Korean History transcribes historical print, it modernizes the spelling. I checked the roughly six hundred thousand colonial-era records they have published, and almost none preserve the older letters. For anyone searching the archive, the original orthography has effectively disappeared.

That makes preservation more than a technical curiosity. A model that keeps the old letters is not simply producing another transcription. It is recovering information that was lost during digitization. For anyone working with pre-1933 Korean print, this may be of importance.

The preservation results only matter if the models can also read the page accurately. On that front, the picture is encouraging. Against four pages of the 1921 magazine Kaebyŏk, where a modern transcription exists, the best current models matched roughly ninety-seven percent of the printed characters directly from the low-resolution scans. That measures reading rather than preservation, since the 1921 transcription already modernizes the spelling. Taken together, though, the results point in the same direction. Current models can read these pages, and on sufficiently old material, the best of them also preserve what makes them historically distinctive.

A note on scale

For a single page, I would use a frontier vision-language model every time.

My own project is different. It involves millions of character crops. Running a frontier model over that volume would incur substantial API costs.

Instead, I built a small voting system with more traditional OCR engines. EasyOCR and a lightweight classifier trained on historical Korean type recognize the Old-Korean, PaddleOCR reads the Hanja, and a simple voting rule chooses between them. The entire corpus finished in about a week on hardware we already owned.

The ensemble has one advantage over a frontier model besides cost. It is explicit about uncertainty. When its engines disagree, it leaves the character blank instead of guessing. A frontier model often does the opposite. It produces a perfectly plausible modern character, and unless you compare it with the original page you may never realize the historical form has disappeared. A blank is an error you can revisit. Silent modernization is much harder to detect.

What this adds up to

For anyone working with pre-1933 Korean print, the practical picture has changed. Current vision-language models can do more than transcribe these pages. The strongest of them can preserve historical orthography that is absent from most digital editions. It can also help one quickly digitize small (or large) amount of works. However for very large datasets cost can quickly skyrocket. It might be worth distilling some of this knowledge into a smaller, more specific model to save on API costs.

That does not mean OCR has been fully solved. The results are model-dependent, page-dependent, and, as far as I can tell, language-dependent. But for this particular corner of the problem, the gap between what was possible a few years ago and what is possible today is surprisingly large.

This is an extension of part of the presentation on using LLMs in research pipelines I gave for Machine Collaborators, series of talks organized by Charles Crabtree (Monash University) where researchers walk through how AI is being used in their work.

OpenRouter recently announced a product called Fusion, which routes a single question through a panel of cheaper models at once and reports that such a panel can come within a point of a frontier model's score at about half the cost. The marketing is new, but the idea a bit less so. Pitting several models against one another has become a standard way to use language models for finding optimal ideas and outcomes. For empirical research, what use might it have?

Subscribe now

I have used a similar setup in two of my own studies. In one, six models read roughly four thousand open-ended survey answers and sorted each into a code, helping me determine what respondents were actually thinking when they navigated a survey experiment. In the other, four models read sixty-seven Korean history textbooks and, using a term extraction technique I created, determined which words best reflected national identity. Sorting answers into fixed codes and discovering which words matter are different problems, but the instrument is the same. It is a panel of open-weight language models, treated as a team of independent coders, voting.

In this piece I reflect on how it works, why I run it mostly on open-weight models, how it relates to the “councils” that Fusion and others have started to build, and what it does and does not support you, as a researcher, in doing. The short version is that it works well, that for coding tasks it holds up against the human labor it replaces, and that its central value is scale rather than certainty.

The method: open models as a panel of coders

Take a judgment a trained research assistant would make, such as reading a piece of text and deciding what it is about, and give it to several models at once, each with the same codebook and the same instructions. Each model receives the same text, the same instructions, and the same codebook, then returns a single labeled decision. A response receives a classification, while the number of models supporting that decision is recorded as a measure of agreement.

Two features of this design matter for the steps that follow. First, I treat each model as a coder. I ask it to assign a label, much as I would ask a trained research assistant, rather than relying on its probability distribution or confidence estimates. (More on this below.) I ask it for a labeled decision, the way I would ask a research assistant, and I read the spread of decisions across the panel the way content analysts read inter-coder reliability, with chance-corrected statistics such as Cohen’s and Fleiss’ kappa. Second, the models vote independently. They run at temperature zero (i.e., no “creativity” permitted), they do not see one another’s answers, and there is no deliberation. Any sense in which the models are played off against one another happens afterward when I line the votes up and read where they part ways.

I run the panel mostly on open-weight models. A proprietary model behind an API can change between versions, and its outputs cannot be reproduced exactly by another researcher. Open-weight models, run locally at a fixed temperature and seed, return the same answers when re-run in the same environment, which is what reproducible measurement requires. In the classification study I still included one proprietary model, GPT-4o-mini, as a convenient reference coder, because it is inexpensive, fast, and unusually capable on the Korean and Traditional Chinese text that those projects involved. The term-extraction panel below is fully open-weight. But GPT-4o-mini is one voice on a panel that is otherwise open and re-runnable, not the authority on it.1

An example of the method. One open-text response from a survey is read by six models that vote independently. The majority becomes the code, and the size of the majority is recorded as a reliability signal.

Three ways to convene a council

The instinct to put models in concert has been built in several distinct ways, and they are worth separating, because they are not after the same thing.

Fusion is the production version. A panel of models answers in parallel, a judge model reads every answer and writes a structured account of where they agree, where they contradict, and what they all missed, and the calling model writes the final answer grounded in that account. What Fusion optimizes is the quality of one answer against its cost, which is why its own documentation says it is not a drop-in replacement and is worth the expense only when a question warrants several perspectives. The models often arrive at different conclusions, but those differences are resolved internally rather than exposed to the user.

Sakana AI’s Fugu uses the same basic approach. Rather than sending every query to a fixed panel of models in parallel, it uses a trained orchestration model to decide whether and when to delegate, select and coordinate models from an agent pool, verify their work, and synthesize the results into a single answer. The system is exposed through one model API, so the complexity of multi-model coordination never reaches your code.

Two other versions come from computer and political science. Andrej Karpathy's llm-council has several models answer, then rank one another with their identities hidden, before a chairman model writes a final answer. Andrew Hall's llm-council-governance extends that logic into an experiment on the decision rule itself, comparing seven governance procedures, from simple majority voting to deliberate-then-vote, on problems with verifiable answers. Deliberation followed by a vote performed best, at 80.9 percent against 71.7 for the strongest single model. There is also an academic Language Model Council in which twenty models write tests, answer them, and grade one another to produce rankings that track human judgment more closely than any single model does.

Three ways to wire a panel of models. All draw on the same lineage. They differ in what they do with the disagreement.

The resemblance is close enough that the model lineups overlap. Hall's council runs Qwen, Llama, Gemma, and Mistral. The classification panel below runs Qwen, Llama, Gemma, and three others. What separates my use from the rest is the last step. Fusion and the councils collapse the panel into a single output, an answer or a decision, and discard the spread behind it. Hall's experiment even finds that deliberation before the vote produces the best single answer, which is the right goal when you want one answer and the wrong one when the spread is the thing you are trying to measure. So I keep the spread, because in a measurement task the agreement, or the lack of it, is itself a quantity of interest. It helps to think of each model as a coder rather than a black box. Each one assigns a label, the panel may agree or divide, and each carries some uncertainty of its own. Those are the qualities I want to read, not average away.

Application one: classification

Take classification first. My thinking on this grew out of my working paper, "What Were They Thinking?", which asks how to validate survey constructs in conjoint designs using the open-text data usually collected only for manipulation checks. A conjoint experiment tells you what respondents chose, but rarely whether they reasoned about the choice the way the design assumes. To find out, I embedded an open-ended question after the conjoint task ("Why did you choose this person?") and coded the answers into a small set of categories. The corpus is about four thousand short answers in Korean and Traditional Chinese, from naturalization experiments in South Korea and Taiwan, where respondents weighed who deserves to be prioritized for citizenship. The codebook the models worked from had five codes: civic commitment, functional integration, economic focus, identity concern, and a none-of-the-above residual.

Six models coded every answer: GPT-4o-mini together with five open-weight models run locally through Ollama, spanning Qwen 2.5 at 3B, 32B, and 72B, Llama 3.1 8B, and Gemma 3 12B. A response is given a classification when at least four of the six agree. Across the corpus the panel’s chance-corrected agreement is a Fleiss’ kappa of about 0.74, which content analysts would call substantial. As more of the six models converge on a label, each model is also individually more confident in it, and the unanimous cases are, likely, the ones a researcher could reasonably trust. The split cases, where the panel divides three against three, are the cases worth handing to a human reader or dealing with in another way.

As more of the six models agree on a code, each model is also individually more confident in it. Most answers sit at high agreement. The split cases are the ones worth reviewing manually.

Two things make this more than a cost-saving trick. The first is scale. Hand-coding four thousand multilingual answers to a publishable standard, with two trained coders and an adjudication protocol, is a substantial project on its own, and it does not scale to the next survey or the one after that. A panel of models codes the full corpus in an afternoon and recodes it with ease, assuming a replicable setup, if the codebook is updated.

The second is quality. I do not treat the models as a cheap approximation of a human coder so much as a fast one that reasons about a short, context-dependent answer in much the way a careful human reader would. Where I had human-coded ground truth and the panel disagreed with it, I read the disputed cases myself, and more often than not I judged the model's label the better one.2 This is consistent with Ryan Briggs and colleagues, who used language models to extract validated data from roughly one hundred thousand political science articles, putting model coding to work at a scale hand-coding could not reach.3

Not every disagreement is the model's mistake to absorb, though. Some are a sign that the codebook is unclear, and those are the ones to chase, because the codebook is the part you actually tune. When I first checked the panel against human-coded ground truth, the agreement was only fair. Reading the misses in a batch, rather than one at a time, showed the models were applying two of my categories the way I had written them and not the way I meant them: one defined too narrowly, the catch-all residual left too broad. I rewrote those definitions, ran the panel again, and looked again, and over two rounds the agreement with the human codes rose from a fair kappa near 0.40 to a substantial one near 0.67. That gain was larger than anything I got from swapping in a bigger model. The model is the coder. The codebook is its instructions, and calibrating the instructions against a few hundred of your own annotations is where most of the accuracy is won.

Application two: finding the words that matter

The second use case approaches the use of multiple LLMs a bit differently. In "Constructing the Nation," written with Aron van de Pol, I study how South Korean history textbooks portray national identity across the postwar decades. This time there is no codebook to apply. The categories themselves are the quantity of interest. We set out to find which words carry the weight of national identity in the corpus, and how their use changes over time.

The standard tool for extracting terms, concepts, or words from a corpus is topic modeling. The standard topic model is Latent Dirichlet Allocation. LDA treats each document as a bag of words and groups terms that co-occur. It is fast and unsupervised, and it is indifferent to context. A word that means one thing in a sentence about colonial resistance and another in a sentence about economic development is the same token to LDA. For a corpus of long, discursive textbook chapters, where the meaning of a term such as minjok (nation, or ethnos) gets its meaning from the sentences around it, the lack of context being taken into account is a real limitation.

An LLM approach handles it differently. Four large models, each from a different country's lab, read the corpus in long passages and propose the identity-related terms they find: EXAONE from Korea, Aya Expanse from Canada, Qwen from China, and Gemma from the United States. A term enters the final set only when at least three of the four propose it, after which each candidate has to clear a corpus-frequency and semantic-similarity check. The models were chosen from different training traditions on purpose, so that agreement across them would mean more than agreement among near-copies. The result is a set of nine defining terms.

Of the nine terms the four-model consensus defines, an embedding-based topic model identifies all nine, while classic LDA identifies five. Context-aware methods find terms that bag-of-words co-occurrence misses.

As a check, we ran an embedding-based topic model, BERTopic, which reads words in context. It identified all nine, with far fewer edge cases than the bag-of-words baseline. BERTopic does not read a textbook whole, though. Its embedding model takes in short passages at a time, because its context window holds only so many tokens.

An LLM faces the same limit, only a much larger one. It cannot take in a whole book at once either, since the text would exceed its context window and be rejected or truncated, but its window is wide enough to read a term against far more of the surrounding passage, and you can window across a long document in overlapping batches. That extra room is where the real difference shows up.

Classic LDA, for what it is worth, identifies only five terms. The panel of LLMs does what topic modeling is meant to do: name the vocabulary that organizes a corpus. But it reads each term in the context of the passage around it, the way a human reader of these textbooks would, and it identifies terms that word co-occurrence alone misses. For a corpus of long documents, that is the difference between a list of frequent words and a list of meaningful ones.

What LLM voting does and does not measure

There are, of course, limits. Cross-model agreement is a measure of reliability, not of validity, and it is not a calibrated estimate of uncertainty. The panel can agree because the answer is clear, and it can agree because the models share training data and therefore share blind spots. The Condorcet jury theorem, which is the formal reason a majority of independent voters grows more accurate as the panel grows, assumes the voters are independent, and language models are not. Their errors are correlated, so a panel of nine carries far less independent information than nine human coders would.4 Agreement should therefore be reported as inter-coder consistency among non-independent coders, rather than accuracy or probability.

This limit is notable, but it is not, in my experience, the main thing. On the one task where I had adjudicated human codes to check against, which came from a separate corpus on North Korean entrepreneurs rather than the citizenship answers above, the models agreed with the human coders at a substantial level, short of the agreement between the humans themselves, and in the cases where they differed I usually sided with the model. The limit imposes modest discipline. Keep some human-coded ground truth in the project, if you can, even a few hundred cases, so that the panel can be checked rather than trusted on faith. Where a within-model confidence is needed, read it from the model's own token probabilities, which come from its predictive distribution, rather than from the agreement count or a model's self-reported confidence.

That last point is worth dwelling on, because if a model is going to stand in for a coder, its uncertainty has to be handled like a coder’s. A careful human coder is not equally certain of every judgment, and neither is a language model. Even with a fixed codebook and temperature set to zero, the model assigns different degrees of confidence to different classifications. That confidence is reflected in the log-probability it assigns to the label it ultimately selects, a quantity derived from the model’s predictive distribution at the moment of the decision. By contrast, a HIGH or LOW confidence tag the model produces if asked is simply another generated response, not a direct measurement of its uncertainty.

I therefore treat the log-probability as an indicator of uncertainty. It records how much probability the model assigned to the label it ultimately selected, according to its predictive distribution. A response that receives a label but a relatively low log-probability is one that merits closer inspection. It captures something the panel does not. The log-probability reflects how certain each model is about its own decision. Agreement across the panel reflects whether different models reach the same decision. The two measures complement one another rather than serving the same purpose.

Is it worth it

It is, if the claim stays within the evidence. A panel of open-weight models is a practical research instrument, and the two applications here show its range. For classification it does the work of a team of coders at a scale no team could match, and codes nearly as well, a little below expert human agreement. For term discovery it does the work of a topic model while reading each term in context, which lets it identify defining vocabulary that bag-of-words methods miss. In both cases the design that makes it trustworthy is the same. Use several models rather than one, choose them to be as different from one another as possible, run them on open weights so the result can be reproduced, and treat a split vote as a flag for human attention rather than as noise to be averaged away.

A panel of LLMs can outperform a single model. But it does not follow that high agreement means the panel is correct. High agreement tells us that these particular models concur, not that they have reached the truth. The two coincide only when the panel is both accurate and sufficiently diverse. Build the panel and pay attention to where it disagrees. But the human stays very much in the loop here, refining the codebook before the run and adjudicating the cases the panel cannot resolve afterward. The panel supplies scale, consistency, and a record of disagreement. The researcher still has to decide what the categories mean, whether the instrument is valid, and when the models and their output should not be trusted.

Using Claude Code

If you want to run something like this yourself, the steps are codified as a set of open-source Claude Code skills I maintain and reach for when working with agents, so each run stays reproducible. The ones closest to this post are model-council-voting, for running a panel and reading its agreement; text-classification, for designing and validating a codebook; and llm-calibration-logprobs, for reading per-decision confidence from token log-probabilities.

Subscribe now