Online panels have long had a quality problem. Some respondents speed through questions (speeding) and some give the same answer to every question (straightlining). Attention checks are the standard way to catch them, but what about survey takers who automate answering? In November last year, Sean Westwood published a paper in PNAS showing that his AI agent could answer online surveys in a consistent persona and pass 99.8 percent of 6,000 common attention checks.1 If that holds, and automated survey completion becomes (or perhaps already is) common, the main tool for catching low-quality respondents no longer works, and any study run on an online sample is compromised.
In August, I fielded survey experiments on attitudes towards immigration in four countries (Germany, South Korea, Singapore and the US), with around 2,500 completes per country from Bilendi (Germany), Rakuten Insight (South Korea and Singapore) and Prolific (US). The core of the survey was two conjoint experiments, in which respondents evaluate a series of profiles whose features are randomly varied. In the first, respondents saw one immigrant at a time and said whether they supported granting that person permanent residence. In the second, they saw an immigration policy that varied in how it was decided and implemented, the area it covered, its direction and the group it affected, and said whether they supported it. Each experiment had eleven tasks, and the last repeated the first exactly. With Westwood’s result in mind, I did not rely on attention checks alone. I added layers of checks for both AI use and low effort. This post looks at the wider problem and then at what I found, or didn’t find, in my own surveys.
What do we know about AI use in surveys?
How large this problem is likely to be is a matter of some debate. Westwood and Frederick flagged 4.4 percent of a sample from Prolific for indicators of using AI (either pasting in text or typing too fast).2 Rothschild et al.’s response to their paper argued that this number was too high.3 They also point out that there are at least three distinct problems. Human respondents may use chatbots to help them answer, simple agents may answer on their own, and autonomous agents may answer from inside a managed panel. The first of these is well documented, and seems quite common. Veselovsky et al. find between a third and nearly half of MTurk workers using large language models (LLMs) to aid them in a summarization task,4 and Zhang, Xu, and Alvero find about a third of their respondents admit to using an LLM to answer open-ended questions.5 The evidence for the other two is thin, with Morning Consult finding signs of OpenAI’s Operator agent in around 0.01 percent of interviews.
There is very little evidence for the last two, but it is possible to check for all three (plus low-effort respondents), so I did.
What I checked
I have grouped the checks by what each was meant to catch. In most cases, the automation checks were run by a short script on four of the conjoint screens (the first task in each experiment, and again for the repeat).
Low effort: Attention checks have long been used to weed out respondents who are not paying attention.6 They are not much help against AI, though, as Operator passed Morning Consult’s attention checks.7 The repeated task gives a baseline of how consistent respondents are with themselves (about 77 percent on average across studies, according to Clayton et al.),8 and I recorded how long each screen took.
Chatbot help: Pasting is a common sign of chatbot use, and disabling copy-paste roughly halved LLM use by crowd workers in one study.9 The script counted pastes. Near the end of the survey I also asked directly, “Did you use an AI tool (such as ChatGPT, an AI browser assistant, or similar) to help you complete this survey?” This is obviously imperfect. An agent built to pass as human would presumably not admit it, while someone using a translation tool might. Still, Xu and Malkin found that most respondents who admitted to using AI had already been caught by a copy-paste check, and that agents usually admitted to being used when asked.10 People do, however, under-report their own AI use.11
Scripts and agents: The script recorded whether anything filled an off-screen “honeypot” text field, which a human cannot see but a script auto-filling every input would fill. Honeypots are inexpensive to add, but Storozuk et al. found none of their bots triggered them,12 and Operator was not fooled by them either. The script also recorded whether the browser reported being under automated control (the webdriver flag, which many automation tools set by default), and it counted mouse movements, clicks, key presses and touches. I could not find a study on whether consumer agents set the flag. Mouse movements say little about respondents on phones. Prolific has found that behavioral metrics of this kind correctly identify agents 95 percent of the time, compared to 60 percent for more classic attention checks.
Platform screening: Qualtrics scored each session with Google’s reCAPTCHA and a fraud and duplicate service, and Prolific runs its own checks to keep its pool authentic.
The automation checks came back empty
Among 10,930 delivered completes, zero participants filled out the hidden field, zero participants used a browser that reported being controlled by an automated program, and zero participants got through all of my measured screens without interacting with any part of the screen. Two participants (both in the US) pasted text. Neither reported using AI. reCAPTCHA flagged 2 to 13 percent of sessions as suspect, but because the reCAPTCHA score is not very specific, and the more direct measurements are all zero, I believe this is just noise. Prolific’s quality checks were similarly unremarkable. Among the 2,462 participants in my main US study, no participant scored Low (problematic) on the LLM check or the bot check, and 19 scored Mixed on the LLM check and 12 on the bot check.
Who admitted to using AI
In total, 854 respondents said they had used AI to complete the survey. This corresponds to 0.4 percent in the US, 4.6 percent in Germany, 8.6 percent in South Korea, and 16.5 percent in Singapore. If these were simple scripts, at least some of them would have set off the automation checks — but none did.
I did some digging to see if I could spot differences between the two groups. People who say they used AI take between a third and a half as much time to complete each conjoint screen in every country (a median of 8 to 10 seconds per screen against 21 to 29). They are also more likely to be using a touchscreen device, fail the attention check, and (in Germany and South Korea) not give the same answer to the same task a second time.
So the “typical” self-reporting person seems to be someone speeding through the survey and using a touchscreen device. If they were using a script I think it would show up in the paradata somehow, but I do not see any evidence for it. There is one small exception. The 11 US respondents who said yes barely moved the mouse (a median of 6 movements per screen against 256), though they also finished each screen in 8 seconds and none of them pasted or set off the automation flag. I do not have any data to confirm this, but it is possible that something about being on a phone makes people more likely to say “yes” (maybe some AI feature on the keyboard or browser?). Translation tools are another possibility, since the rate was highest in Singapore, where some respondents may have run the questions through a translator. I gave Singaporean respondents the option to take the survey in English or Mandarin Chinese (most opted for English), so I’m not sure about the translator explanation.
The self-report also overlaps with the checks I already had. In Germany, South Korea and Singapore, between 28 and 36 percent of those who said yes failed the attention check, against 3 to 8 percent of those who said no. Between 36 and 40 percent showed two or more of four low-effort signals (a changed answer on the repeated task, speeding, or identical answers across profiles in either experiment), against 8 to 13 percent of everyone else. So saying yes goes with low effort, but most of the people who said yes did not show two or more of these signals.13
How each panel handled self-reported AI use
The self-report was still used for billing. With both Bilendi and Rakuten, I agreed to replace a self-reporter if they met at least two of four criteria. These were (i) saying something different for the repeated task, (ii) being faster than half the median time, (iii) answering the same for all profiles in the first experiment, and (iv) answering the same for all profiles in the second experiment. Self-reporters who met none or only one of the criteria were kept: 79 in Germany, 150 in South Korea and 286 in Singapore. I approved all my Prolific completes (Prolific has no pool to replace them from), and I only dropped US participants for the pre-registered reasons. The attention check accounted for more replacements than the AI rule in all three countries. Replacement means the panel did not bill me for the respondent and recruited a new one in their place. I still have the data on everyone who was replaced.
This is worth knowing for anyone running a study like this. Prolific is automated, so I never discussed any of this with anyone from the company — although they seem quite responsive and if I had major concerns, I turst I would have been able to get in touch with someone involved with quality control. The other vendors were very interested in all of this and willing to look into any respondent I suspected of being low quality — for whatever reasons, AI or not. Of course, sample quality is central to their business model. You can typically work with your project manager to make sure you end up with the best-quality sample possible.
What the checks can and cannot catch
In these four online panels in the summer of 2026, I found no measurable sign of automated responding. The problem that did show up was low effort, from respondents who sped through, straightlined, or answered the repeated task differently.
So, I think I collected generally good-quality samples. After doing the replacements, I found that when I showed the same task again, respondents answered the same as the first time 78 to 80 percent of the time in Germany, South Korea, and Singapore, and 89 percent of the time in the US. This is at or above the 77 percent average in Clayton et al. None of my automation checks flagged anyone, and I only had to replace 5 to 13 percent of completes.
The checks I used I think work well to detect crude forms of automation, but they would not catch someone running an AI assistant in their browser, as the AI could generate normal browser events for all input. If the AI agent was sophisticated enough, it would see the page just as a human does and would not fill the hidden field. Mouse movement might help here for now, as someone using a computer had a median of about 250 mouse movements per screen (in US respondents answering no to my AI question), but most non-US respondents were on touchscreens. These measures only catch the kinds of automation I thought to look for. There could be others slipping through undetected.
Insofar as lessons go, I recommend using layered checks. In particular, I recommend using behavioral checks on a repeated screen, and then if a respondent answered yes to the AI question, deciding whether to replace them based on those behavioral signals.
So is the attention check compromised? Perhaps, if someone is trying to defeat it. However, I found no sign of anyone doing that in my samples. People simply rushed through the survey, and a repeated task, a clock and the attention check were enough to catch them. Whether or not my findings from my summer surveys can be relied upon going forward is a big open question. I think probably not. Browser agents are getting cheaper and better at acting like people, and the checks that came back empty here are the first ones they will learn to pass — if they aren’t already passing them. The case for running them anyway is they are easy to implement and cost you basically nothing. You can build these into, for example, a Qualtrics AI design skill, taking into account any new contributions that people have made. And perhaps in collaboration with your vendor, who has the strongest interest in the world (their business model) to ensure that their samples are human and that their survey takers are not automating the process.
Sean J. Westwood, “The potential existential threat of large language models to online survey research,” Proceedings of the National Academy of Sciences 122, no. 47 (2025): e2518075122, https://doi.org/10.1073/pnas.2518075122.
Sean J. Westwood and Samuel Frederick, “Reply to Van der Stigchel et al.: Empirical evidence that AI survey contamination is real and substantial,” Proceedings of the National Academy of Sciences 123, no. 8 (2026): e2537420123, https://doi.org/10.1073/pnas.2537420123.
David M. Rothschild, Soubhik Barari, Trent D. Buskirk, Andrew Gordon and D. Sunshine Hillygus, “Reply to Westwood: Questioning the empirical evidence that AI survey contamination is real and substantial,” unpublished letter, 2026, https://researchdmr.com/files/Letter_Westwood.pdf. Gordon works for Prolific.
Veniamin Veselovsky, Manoel Horta Ribeiro and Robert West, “Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks,” arXiv:2306.07899 (2023), https://arxiv.org/abs/2306.07899.
Simone Zhang, Janet Xu and AJ Alvero, “Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization,” Sociological Methods & Research 54, no. 3 (2025): 1197–1242, https://doi.org/10.1177/00491241251327130.
Adam J. Berinsky, Michele F. Margolis and Michael W. Sances, “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys,” American Journal of Political Science 58, no. 3 (2014): 739–753, https://doi.org/10.1111/ajps.12081.
James Martherus, Alexander Podkul, Edgar Cook and Robert Liebowitz, “How to Detect AI-assisted Interviews in Online Surveys,” Survey Practice 18, no. 1 (2025), https://doi.org/10.29115/SP-2025-0016.
Katherine Clayton, Yusaku Horiuchi, Aaron R. Kaufman, Gary King and Mayya Komisarchik, “Correcting measurement error bias in conjoint survey experiments,” American Journal of Political Science (2026), https://doi.org/10.1111/ajps.70076.
Veniamin Veselovsky, Manoel Horta Ribeiro, Philip J. Cozzolino, Andrew Gordon, David Rothschild and Robert West, “Prevalence and Prevention of Large Language Model Use in Crowd Work,” Communications of the ACM 68, no. 3 (2025): 42–47, https://doi.org/10.1145/3685527.
Zane Xu and Nathan Malkin, “A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents,” arXiv:2607.00403 (2026), https://arxiv.org/abs/2607.00403.
Yier Ling, Alex Kale and Alex Imas, “Underreporting of AI Use: The Role of Social Desirability Bias,” in Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (2026), 1–22, https://doi.org/10.1145/3772318.3791073.
Andie Storozuk, Marilyn Ashley, Véronic Delage and Erin A. Maloney, “Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks,” The Quantitative Methods for Psychology 16, no. 5 (2020): 472–481, https://doi.org/10.20982/tqmp.16.5.p472.
Each experiment also ended with a manipulation check, which asked whether respondents had noticed differences in the attribute being tested (how the immigrant entered the country, and how the policy was decided). In Germany, South Korea and Singapore, 20 to 35 percent of respondents who failed the attention check or showed two or more low-effort signals said they had not noticed at least one of these, against 9 to 18 percent of those with no flags. Respondents who said they used AI went the other way. Only 9 to 10 percent of them missed one, against 15 to 21 percent of those who said no. I do not have a good explanation for this, though the manipulation check is itself a self-report.
nice write up -- if you're interested in turning this into a quick academic paper could be a good fit for our special issue https://kevinmunger.substack.com/p/an-experiment-in-ai-peer-review