This is an extension of part of the presentation on using LLMs in research pipelines I gave for Machine Collaborators, series of talks organized by Charles Crabtree (Monash University) where researchers walk through how AI is being used in their work.
A recent product from OpenRouter called Fusion has multiple "cheap" models answer the same question, which they say gets within 1 point of the best "frontier" model at about half the cost. This isn't a novel idea, but the marketing is new. Pitting several models against one another has become a standard way to use LLMs. So what use might it have for empirical research?
I’ve used a similar setup in two of my studies. In one, six models read about 4,000 open-ended answers to a survey question and sorted them into pre-defined codes, helping me to understand what people were thinking when they answered a survey experiment. In another, four models read 67 history textbooks published in South Korea and (using a technique that I developed) identified the set of words that best described how the textbooks related to national identity. The research questions and coding schemes are different, but the empirical instrument is the same. In both cases, I used a panel of open-weight language models, treated as independent coders, to vote.
Some thoughts on how this works, why I run mostly open models, how it relates to the “councils” being explored by Fusion and others, and what this does and doesn’t do for research. It works really well, actually. In terms of coding tasks it is on par with spending human hours on it. Its value, however, is very much about scale rather than certainty.
The method: open models as a panel of coders
Given a task where one may traditionally hire a trained research assistant (e.g. reading a piece of text and labeling what it is about), provide the same codebook and instructions to multiple models. Each model reads the text, is given the same instructions and codebook, and provides a single response (i.e. the label). The majority label is the classification, and the number of models that agreed on it is recorded as a measure of agreement.
Two features of this design are worth noting before I move on to describe the steps taken. First, the models are treated as coders. As such, they are tasked with assigning a label in much the same way a trained research assistant might. This stands in contrast to an approach that might be interested in the probability distributions provided by the models or any other measure of confidence (more on that below). Simply put, I ask the models to make a decision and to provide that decision “labeled” in a way that I can compare them to one another. Just as human coders can be compared by calculating (chance-corrected) inter-coder reliability (e.g., Cohen’s and Fleiss’ kappa), so too can the LLMs. Second, the voting is conducted independently. Not only are the models run at “temperature” zero (i.e., they are not permitted any “creativity”), they are also not permitted to view one another’s answers or otherwise deliberate. The models are still played off against one another, just at a later stage, when I line up the votes and look at where they part ways.
I run most panels on open-weight models because proprietary models behind API calls can change over time, and can't give repeatable answers if run elsewhere. Open weights models can be run locally with a fixed temperature and seed, and provide the same answers when run again in the same environment, making them suitable for collecting reproducible measurements. The model panel for the classification study includes one proprietary model, GPT-4o-mini, because it's a very cheap, fast, and capable reference coder that also has good performance on Korean and Traditional Chinese, which are relevant for those projects. The panel for the term extraction below is fully open weights, however, and GPT-4o-mini is only one voice in a larger open panel.1
An example of the method. One open-text response from a survey is read by six models that vote independently. The majority becomes the code, and the size of the majority is recorded as a reliability signal.
Three ways to convene a council
There are a few ways to use models in concert, and they aren't all after the same thing.
Fusion is, I think, the production version of this. The way it works is that a set of models answer in parallel, then a judge model reviews the outputs and produces a structured summary of consensus, disagreements, and missed items, then a calling model uses that summary to produce the final answer. It’s designed to optimize for answer quality at a given cost, but is not intended as a drop-in replacement. It is intended for cases where asking for multiple perspectives is valuable. The models often do come to different conclusions, but these differences are resolved in a way that is not exposed to the user.
Sakana AI’s Fugu uses a similar approach, with a trained orchestration model that selectively delegates work to its pool of agent models, verifying and synthesizing their results into the final answer. All this multi-model coordination happens under the hood, exposed to you as a single model API.
There are also two other related projects that I found from the computer and political science angle. First, there is Andrej Karpathy's llm-council, where multiple language models answer a question, then rate each other's answers while hiding the identity of the model, and then the "chairman" model answers the question using the ranked answers. And then there is Andrew Hall's llm-council-governance, which studies how the "decision rule" affects the outcome. He tries out 7 different governance rules, such as majority voting or deliberate then vote, using problems with known answers. The best performing governance structure was deliberate then vote, with an accuracy of 80.9%, compared to 71.7% for the strongest individual language model. There is also the more "academic" Language Model Council, where 20 language models write tests, answer them, and grade each other's answers. The authors show that the models can rank each other in a way that is more closely aligned with human rankings than any individual model.
Three ways to wire a panel of models. All draw on the same lineage. They differ in what they do with the disagreement.
These are similar enough that the models involved in the respective lineups overlap. Hall asks a council of Qwen, Llama, Gemma, and Mistral, and the panel that classifies below includes Qwen, Llama, and Gemma, along with three other models. But where this differs is that for Fusion and the councils, the ultimate goal is to distill the panel’s answers down to a single one, and in Hall’s case the best approach (deliberate then vote) is the one that produces the best single answer. In contrast, here I want to retain the spread of answers, because the spread is what I’m measuring. I view each model as a coder who is assigning a label to an observation, and I want to know if the coders agree or disagree, and how uncertain each coder might be.
Application one: classification
I will begin with classification. It is tied to a working paper I am writing titled “What Were They Thinking?” where I seek to find a way to use open-ended text (usually collected only as a manipulation check) to validate the constructs in a conjoint experiment. A conjoint tells you what respondents chose, but rarely whether they reasoned about it the way the design assumes, so I added the open-ended prompt “Why did you choose this person?” after the conjoint task. What I have are about 4,000 short answers from conjoint experiments on naturalization priorities. I have two corpora, one in Korean (South Korea) and the other in Traditional Chinese (Taiwan). My codebook contains 5 codes, including a “none of the above” code, and the codes focus on civic commitment, functional integration, economic focus, and identity concerns.
I code every answer with 6 LLMs: GPT-4o-mini, and five local Ollama models (Qwen 2.5 3B, 32B, 72B, Llama 3.1 8B, Gemma 3 12B). I assign a classification when at least four of the LLMs agree on that classification. It gives a Fleiss' kappa of ~0.74, which in content analysis terms is "substantial" agreement. Moreover, when more of the LLMs agree on a classification, each LLM is more confident in that classification - so when all six LLMs agree on a classification, it's probably safe to trust that classification. I deal separately with cases where the LLMs split 3-3 (either by human coding or some other way).
As more of the six models agree on a code, each model is also individually more confident in it. Most answers sit at high agreement. The split cases are the ones worth reviewing manually.
Two things make this more than a cost-saving trick. First, scale. Coding 4,000 answers to publishable standards, in multiple languages, with two trained coders and an adjudication protocol, would be a major project, and the next survey would be another one. It took the panel of models an afternoon to code the entire corpus. It could be done again, if set up to be replicable, and if the codebook is revised, with no trouble at all. It scales.
Second is quality. Rather than thinking of the models as a cheap way to essentially replicate the work of a human coder, think of them as a fast way to have someone reason through a particular answer to a particular question, given a particular context, in this case, a relatively short one. I did look at the ones where there was actual ground truth in the form of human coders and the panel disagreed, and usually I thought the label the model produced was the more appropriate one.2 Others have found similar results: check out the work by Ryan Briggs and others where they use language models to extract a lot of validated data from ~100,000 political science articles.3
I do want to note that not every disagreement is the model's mistake. Some show where the codebook could be clearer, and those are the useful ones to look at. You're ultimately tuning the codebook, and if your goal is accuracy, you should make sure the panel roughly matches whatever ground truth you have (i.e., human codes). When I first ran this, I got fair agreement with the human codes; looking through a batch of those, I saw that the models were following the codebook definitions for two categories, but those definitions didn't match what I had in mind (one too specific, a residual category too general). I updated the codebook definitions and re-ran the panel a couple times, and I was able to get agreement between the panel and the human codes from fair (kappa = 0.40) to substantial (kappa = 0.67), which is a bigger improvement than I got from switching models. Remember, the model is the coder, so you're essentially calibrating your coder with the codebook against however many human codes you have, and that is where most of the accuracy comes from.
Application two: finding the words that matter
Second, I will discuss a slightly different approach to using multiple LLMs. In Constructing the Nation (with Aron van de Pol) we examine how the representations of national identity in South Korean history textbooks have changed over the postwar period. However, in this project, there is no codebook that we apply. This is because the categories that we want to code for are the quantity of interest. Thus, we need to identify the words that represent national identity in the corpus, and then how the use of these words has changed over time.
If someone wanted to extract terms, or concepts, or even just words from a corpus, the standard method would be to use a topic model. And the standard topic model to use would be Latent Dirichlet Allocation. LDA, though, treats each document as a bag of words. It works by grouping words that appear together in documents. It’s very fast, and it doesn’t need supervised data, but it doesn’t take context into account at all. That can be a problem if the meaning of a word (especially if it’s a word like minjok, meaning something like nation or ethnos) depends a lot on the context it appears in, which is very often the case in the long, discursive chapters of the textbooks I’m working with.
Another approach using LLMs that I tried consisted of reading long parts of the corpus with 4 diverse LLMs and then asking them to propose terms associated with identity. If at least 3 of them proposed a term, then this term was added to a set of terms, if and only if it passed a frequency and semantic similarity test on the corpus. The 4 LLMs chosen were from 4 different countries in order to maximize their diversity. They were EXAONE (Korea), Aya Expanse (Canada), Qwen (China), and Gemma (USA). This method found 9 terms that defined identity.
Of the nine terms the four-model consensus defines, an embedding-based topic model identifies all nine, while classic LDA identifies five. Context-aware methods find terms that bag-of-words co-occurrence misses.
To check, we ran BERTopic, a topic modeling technique based on embeddings, and which, unlike a bag of words approach, does read the context of words. BERTopic also found all 9 terms, and with far fewer edge cases. In fairness, BERTopic does not read the whole textbook: the embedding model it uses only reads short passages at a time, because it can only handle a limited number of tokens at once.
An LLM hits the same limit, just a bigger one. If you were to copy an entire book into the LLM it would either reject the prompt or cut it down because it would be too long. But its window is wider, so it can read more context around a term. It can also read documents in large batches with overlap to cover the entire document.
As a side note, the ‘Classic’ LDA only finds 5 terms. The panel of LLMs does what topic modeling should be doing, finding the ‘important’ terms in a corpus. They do this in the context of the passage, whereas classic topic modeling only looks at word occurrence. For a corpus of long documents, this makes a big difference.
What LLM voting does and does not measure
A few caveats: this is just measuring inter-model agreement, not validity, and this is not a well-calibrated uncertainty estimate. The models are not independent (they are likely trained on a lot of the same data), so the guarantees of the Condorcet jury theorem don’t apply. If anything, I’d interpret this as a measure of inter-coder consistency between non-independent coders. (So, I’d expect 9 models to be less informative than 9 humans, for instance.)4
That’s worth remarking on, but I don’t think it’s the main thing here. Relative to the one set of human-adjudicated codes I had (a separate corpus, about entrepreneurs in North Korea, not the above), the models agreed with the humans quite a bit (not as much as the humans agreed with each other, but still a lot; and in the cases where the models and the humans disagreed, I usually sided with the model). The limit does impose a decent bit of discipline, though. If you can, I’d recommend including some human-coded ground truth, even if just for a few hundred cases, in any given project. If you want within-model confidence, you should probably derive that from the model’s token probabilities (i.e. its predictive distribution) rather than from the agreement count or the model’s self-reported confidence.
This last point is important, because if the language model is used as a substitute for a coder, its uncertainty should be handled like a coder’s. Even if the codebook is fixed and the temperature is zero, the language model will be more or less certain about its classification choices. This shows up in the log probability of the classification that the model chose, based on its predictive distribution for that classification. It is not the same as having the model also answer whether it is “HIGH” or “LOW” confidence in its classification choice, because that is simply another generated answer.
I am using log prob simply as a measure of uncertainty. It tells me how much probability the model assigned to the label it chose, according to its predictive distribution. The responses that are labeled, but with a low log prob, are what I am interested in looking at more closely. This is something that the panel does not give me. The log prob is a measure of how certain each of the models is, and the panel agreement is a measure of whether they are all making the same decision.
Is it worth it
It is, if the claim stays within the evidence. Here, I explored using a panel of open-weight models for research. Across two applications, it looks like this is a very useful tool. First, I show that a panel of open-weight models can classify texts just slightly worse than the agreement of human coders. This means that a panel can plausibly be used as a replacement for large numbers of human coders. Second, I show that a panel can also discover terms in a corpus; it outperforms standard bag-of-words topic models by leveraging contextual information to identify meaningful, defining terms. Taken together, these results illustrate the benefits of using open-weight, diverse panels of LLMs to ensure replicability and the value of analyzing disagreement.
A panel of LLMs can be better than a single LLM. However, when the LLMs agree, that doesn't mean they are right; it means the particular LLMs in the panel agree. The two only line up if the LLMs are both accurate and diverse. Despite this, I think it is worth building a panel and paying attention to where it disagrees. (Note that this approach still requires lots of human-in-the-loop, both pre and post LLM run, to refine the codebook and adjudicate cases.)
Using Claude Code
If you want to do something like this, the most relevant of my open-source Claude Code skills (codifications of steps I follow when using agents, to improve how reliably I can get them to do things) are model-council-voting (for running a panel and seeing how they agree), text-classification (for designing and validating a codebook), and llm-calibration-logprobs (for getting the model’s confidence in each of its decisions, in terms of log probabilities of the tokens it emitted in making that decision).
The full classification panel is GPT-4o-mini plus five open-weight models run locally via Ollama at temperature zero with a fixed seed: Qwen 2.5 (3B, 32B, 72B), Llama 3.1 (8B), and Gemma 3 (12B). The case for running open weights rather than depending on a proprietary API is reproducibility: API models drift between versions and cannot be rerun deterministically (Barrie et al. 2025). GPT-4o-mini earns its place as a reference coder on grounds of cost and strong multilingual performance, not authority, and agreement with it rises monotonically with open-model size (Qwen 2.5: 0.73 at 3B, 0.83 at 32B, 0.86 at 72B), which is a useful sanity check on the smaller models.
The check comes from a separate corpus coding why North Korean entrepreneurs make the choices they do, where two trained humans coded every response. The two humans agree at kappa 0.88. Each model agrees with the humans at about 0.61 to 0.67, roughly what one would expect of a competent third coder. The models are also overconfident against human codes when their token probabilities are read as calibrated estimates (expected calibration error of 0.15 to 0.26, against 0.01 to 0.10 when scored against the panel’s own majority), which is the technical form of the point that agreement is not the same as accuracy.
Briggs, Mellon, and Arel-Bundock, “It must be very hard to publish null results” (working paper). They code a large corpus of articles for whether results are null, using LLMs validated against human coding, and document a steep gap between the share of published abstracts reporting non-null results and the share reporting nulls.
A recent study, “Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels” (Kohli 2026), estimates that a panel of nine frontier models from seven families carries only about two independent votes’ worth of information, because the models tend to be wrong on the same items. The remedy is real diversity among the voters, which is the reason the term-discovery panel draws its four models from four different national labs.