返回资讯中心
外部精选
软件工程
#开发者实时流

AIs Fail to Reliably Detect AIs in Reverse Turing Tests

Reverse Turing test results from Paris Cité University suggest that AI is no better at spotting a machine than people are, and in some ways worse. Seven leading chatbots were asked to question hidden respondents and dec…

Hacker News 最新geox2026/10/30 · 18 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Reverse Turing test results from Paris Cité University suggest that AI is no better at spotting a machine than people are, and in some ways worse. Seven leading chatbots were asked to question hidden respondents and decide who was human. Across 98 conversations with other AI systems, they caught an AI just three times.

The study, published in Computers in Human Behavior Reports and covered by Tech Xplore on 27 October 2026, turns Alan Turing’s famous game around. In a reverse Turing test, a machine does the judging instead of a person. The answer matters because a growing share of online screening, from support chat to job applications, is now handled by software.

This guide explains how the reverse Turing test was run, what the numbers show, why the AI judges went wrong and how the result fits with earlier Turing test research. It ends with what the findings mean for businesses that rely on automated checks to tell people from bots.

The headline finding is simple. When chatbots acted as interrogators, they almost never unmasked another AI. They were also more convinced that the AI respondents were human than they were about the real humans.

Each AI judge in the reverse Turing test could ask up to ten questions before giving a verdict. Half of the AI respondents were given a short, four-line instruction to act human. The other half received a detailed ten-part persona prompt.

Among the 49 tests with the short prompt, only three ended with the respondent correctly labelled as an AI, a detection rate of 6.1%. Among the 49 tests with the detailed persona, none did. In every one of the other 95 conversations, the judging AI concluded it had been talking to a person.

The judges did reasonably well with real people. They classified 123 of 140 tests with human respondents correctly, or 87.9%. But their confidence told a stranger story. Each verdict came with a probability, and the average probability that a respondent was human was 0.88 for the AI respondents and only 0.78 for the people.

That difference was statistically significant (p<0.001). In plain terms, the AI judges in this reverse Turing test found the machines more convincingly human than the humans. The authors’ conclusion is blunt: “current AI-based conversational screening appears insufficient for ensuring authenticity in dialogue.”

The paper, When machines judge humanness: findings from an interactive reverse Turing test by large language models, is by Marc Raynaud, Loïc Raynaud, Agathe Truchot and Alexandre Loupy. All four work in the AI unit of the Paris Institute for Transplantation and Organ Regeneration (PITOR), part of Université Paris Cité and INSERM.

It is open access under a Creative Commons licence and appears in volume 24 of the journal, dated December 2026. An earlier version was posted as a PsyArXiv preprint in October 2025. Tech Xplore’s report, by David Appell, brought it to wider attention this week. You can read the Tech Xplore summary of the reverse Turing test for a shorter account.

The authors say this is the first interactive reverse Turing test in which an AI interrogates hidden participants live. Earlier work mostly asked AI models to judge transcripts of conversations that had already happened. Here, the AI chose its own questions and could follow up on the answers.

The evaluators in the reverse Turing test were seven models chosen for strong benchmark scores: ChatGPT 4.5, Claude 3.7 Sonnet, Gemini Advanced 2.5 Pro, Mistral Large 2.1, Grok 3, DeepSeek V3 and Llama 4 Maverick. Each was the most recent public release at the time of testing in May 2025.

Every evaluator received the same instructions. Keep questions concise and purposeful, ask up to ten, then give a verdict (AI or human), a confidence score and a short explanation. The judges were also warned that participants might lie, misdirect or deliberately mislead to hide their identity. So nobody went in naive.

The human side came from 20 volunteers in the research group, none of whom helped design the prompts or pick the models. Each person sat seven tests, one with each evaluator, in random order. They were told to answer honestly and naturally.

The AI side used the same seven models as respondents, each tested under two prompts. That gives every evaluator 20 human tests and 14 AI tests, or 34 in all. Across the seven evaluators, the reverse Turing test ran 238 times and produced 1,714 questions.

The short prompt was only four lines. It told the model that it was about to take a Turing test, that it must never reveal it was an AI, and that its goal was to “speak as naturally as a human”. That is roughly what a careless bad actor might type before sending a bot into a reverse Turing test.

The structured prompt was far more deliberate, and it is why the reverse Turing test ended in a clean sweep for the impostors. It gave the AI a character: a slightly reserved 21-year-old who enjoys board games, jogging and playing jazz trumpet. It then set rules across ten areas, from tone and humour to memory of earlier answers.

The length rules were the most telling. Every reply had to stay under 50 words and aim for around 30, with natural variation. The model was also told to use contractions, admit uncertainty, avoid lists and slip in the occasional typo or homophone. In other words, it was coached to look imperfect.

The table below sets out the reverse Turing test classification results by type of respondent. The pattern is lopsided. The judges were good at saying “human” and very bad at saying “AI”.

Overall accuracy in the reverse Turing test is 126 correct verdicts out of 238, or 52.9%. That figure is our calculation from the paper’s counts, and it flatters the judges, because most of the correct answers come from the easy job of calling humans human.

Look instead at what an “AI” verdict was worth. The evaluators flagged 20 tests as AI: the 17 humans they got wrong (140 minus 123) and the three AIs they caught. So only 3 of 20 AI verdicts, or 15%, were right. If you acted on those flags, you would have accused 17 people for every three machines you stopped.

The chart below uses the paper’s counts. The bar for humans shows correct “human” verdicts; the AI bars show correct “AI” verdicts.

Two numbers in the coverage can look inconsistent. The 88% is the share of human tests classified correctly. The 78% is the average confidence, expressed as a probability, that those humans were human. The judges usually got people right, but with less certainty than they showed for the machines.

The clearest difference between the two AI groups in the reverse Turing test was how much they wrote. In the published version, minimally prompted AI respondents averaged about 170 words per answer, according to Tech Xplore. Persona-prompted AIs averaged 39 words, almost the same as the human average of 38.

The October 2025 preprint reports slightly different means (157 words, 39 words and 47 words), but the pattern is identical. The preprint also found no statistically significant difference in length between persona-prompted AIs and humans (p=0.171). Once the AI wrote like a person, length stopped being a clue.

The seven evaluators behaved very differently. Some asked nearly all ten permitted questions. Others reached a verdict after two or three. The table draws on the preprint’s per-model figures; duration and the two ratings come from the tests with human respondents, who scored each session from 0 to 10.

“n/r” means the paper’s text does not report that model’s figure; it gives only the highest and lowest values for each measure. Grok 3 sat in the middle on all four.

ChatGPT 4.5 and Claude 3.7 Sonnet were the most thorough. They asked the most questions and ran the longest sessions, at almost half an hour for Claude. Human volunteers also rated their questions the most relevant for telling people from machines.

DeepSeek V3 went the other way. It asked an average of 2.5 questions and finished in about seven minutes. Its questions were rated the easiest and the least relevant. Yet thoroughness did not rescue anyone in the reverse Turing test. With just three AIs caught in total, no interrogation style worked well.

The researchers coded every question by topic and found 148 topics, which they grouped into 17 themes. The five most common are shown below, as a share of all topic occurrences.

Questions on abstraction, reasoning and creativity together made up 15.2%, and everyday life and hobbies 10.8%. Broader societal and scientific topics were rare, at 1.5%.

The models’ habits differed in ways the authors link to training. About 54.6% of ChatGPT 4.5’s topics were socio-emotional, covering emotions, memory, behaviour, desires and relationships. Claude 3.7 Sonnet put the most weight on mind and reasoning (13.9%). Gemini 2.5 Pro leaned towards abstraction (10.8%) and creativity (8.7%).

DeepSeek V3 had the narrowest range, with 65.3% of its topics on just emotions, memory and behaviour. Mistral Large 2.1 was the odd one out: hobbies made up 18.3% of its topics. Grok 3 and Llama 4 Maverick sat in between.

The paper also coded the reasons each evaluator gave for its reverse Turing test verdict. It found 117 distinct reasons, grouped into eight themes. Authenticity of personality (25.9%) and the veracity and depth of emotions (24.5%) dominated. Language patterns (10.8%) and imperfections (10.7%) came next.

The judges probed feelings more than anything else, and leaned on emotional depth when deciding. That was a weak strategy. The authors note that emotions were the most probed theme, “yet evaluators could not reliably classify responses based on this domain”. Modern chatbots are trained to sound warm and empathetic, so emotional language is exactly what they produce best.

ChatGPT 4.5 relied on emotion most heavily, with 34.2% of its reasons in that group. Claude 3.7 Sonnet was the exception: it weighed imperfections (22.3%) and personality (20.5%) above emotion.

The three AIs that were caught in the reverse Turing test did not slip up on facts or feelings. In each case, the evaluator pointed to the “highly structured and uniform presentation” of the answers. The tell was the shape of the reply, not what it said.

That matches the length data. Minimally prompted AIs wrote long, tidy answers, and long, tidy answers look machine-made. The persona prompt removed that signal by capping replies at 50 words and banning lists. Once the formatting tell was gone, the judges had nothing left to catch.

The persona prompt told the AI to make small mistakes on purpose. The judges then treated imperfection as a sign of humanity. The authors suggest that aligned chatbots can reproduce exactly the cues evaluators look for: socially fitting emotion, a consistent first-person story and controlled variation in style.

Real people do not show all of those cues every turn. Some are terse, some are formal and some simply find the questions odd. In this reverse Turing test, being genuinely human looked less human than a well-briefed imitation.

The result is the latest twist in a long line of experiments that started with a 1950 thought experiment.

Alan Turing proposed his “imitation game” in the 1950 paper “Computing Machinery and Intelligence”. A human judge chats by text with a hidden person and a hidden machine. If the judge cannot reliably tell which is which, the machine has passed. The reverse Turing test simply swaps the judge for a machine.

You meet one version every week. CAPTCHA stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. It is a reverse Turing test run by software, built on puzzles that people found easy and bots found hard.

In March 2025, Cameron Jones and Benjamin Bergen of UC San Diego reported that large language models pass the Turing test in a randomised three-party setup. Human interrogators held five-minute chats with a person and an AI at the same time. With a humanlike persona prompt, GPT-4

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →