Can Machines Follow One Voice Through a Crowded Room?

I’ve been noodling on ideas for reliably distinguishing humans from machines, and went down a rabbit hole leading to Moravec’s paradox: the idea that certain tasks which are trivial for humans are difficult for computers, because human abilities have been honed over millions of years via evolution. That led me to pose the question: computers can transcribe speech, but can they pick out and understand one voice in a loud, crowded environment? I decided to test it out.

Selective auditory attention is the trick human brains use to listen to a speaker at a loud cocktail party. If humans can do this easily and computers can’t, that’s a neat reverse Turing test. For the experiment, I generated a corpus of synthetic audio clips such as the following to run through a suite of speech recognition models:

From 40 synthetic cocktail-party-like audio clips tested against 8 speech recognition model configurations compared with 146 human answers, humans performed roughly as expected, and machines performed far better than expected.

Code & data for this article can be found at github.com/cloudkj/crosstalk

My goal was to test two aspects of selective auditory attention. First, whether human success rates for picking out speech in chatter will be high and machine speech recognition success rates will be low. As rough targets, I picked over 80% for humans and under 20% for machines; these are simple gut-feel numbers for what could constitute a useful test, but are not derived from anywhere.

Second, if the gap between human and machine performance, however narrow or wide, holds up across a variety of speech recognition models spanning different levels of capacity and complexity.

A 1953 paper from E.C. Cherry was the original work that introduced the cocktail party problem, and highlighted the human ability to selectively focus on a single auditory message while tuning out background noise. A number of previous experiments and analyses in the modern era were also discovered in the literature, a number of which do not support the core hypothesis. Previous papers as early as the late 2000s up to early 2020s showed the ability of automatic speech recognition (ASR) models to break audio CAPTCHAs.

A recent study from researchers at PeopleTec (Noever & McKee, 2025) covered a number of efforts related to selective auditory attention, and their takeaways with regard to overlapping speech recognition were directionally promising; average model performance achieved 17.9% compared with 61% baseline human accuracy, specifically for the task of identifying overlapping speech. On the other hand, the best performing model in that study (GPT-4o) performed roughly on par with humans. However, the audio generation methods were unspecified and no underlying datasets or code were available, and so this study mostly served as a reference point.

The task at hand was to devise a method to produce audio that reliably exhibits a clear gap in performance on the selective auditory attention task between humans and speech recognition models. In short, I had to generate overlapping speech clips then measure model task performance. I went through multiple rounds of audio-mix tuning in order to generate synthetic audio clips that were intelligible to my ears and simultaneously plausibly hard for machines. Once the audio synthesis seemed sufficient, I gathered actual transcription responses from human participants that listened to these clips which were compared against transcription results from models.

An initial proof-of-concept was built using espeak-ng to flesh out the end-to-end pipeline and provide a demo for a small number of human testers. The first version used simple two-word phrase pairs, along with various adjustments on pan, distractor, and stagger parameters. The initial model comparison baseline was the lightweight Whisper base model.

These first clips turned out to be largely unintelligible, and difficult to transcribe for both models and humans alike. In the next version, I switched to full sentences with multiple overlapping voices and decibel adjustments. Qualitatively, these sounded more like plausible speech, but the underlying voices still sounded too artificial; the background chatter was also too garbled to resemble an actual cocktail party environment.

Ultimately, a full parameter sweep produced a number of variants which were successful at producing speech that was clear enough for my ears, using interjection fragments with a handful of voices, along with a low-pass muffle. However, the resulting clips turned out to also be easier for the models, in which Whisper base was able to achieve 90% pass rate based on simple word overlap.

In the next round, I switched to more distinct, human-sounding, natural voice sources using Piper TTS as a way to address robotic voices with unexpected specific accent variations. I landed on a stable candidate to produce discernible audio with an initial 10% model success rate (Whisper base on word overlap), using a fixed target voice and a pool of five babble voices. The babble voices spoke randomly selected sentence fragments from a pool of filler text, and used 200 ms gaps with jittered start delays of up to 800 ms per background speaker. Subsequent tuning was applied to make the audio clips more qualitatively tractable to my own ears.

Ultimately, I ended up with a corpus of 40 target sentences (e.g. “the old man walked slowly to the store”) combined with a filler pool of 22 sentence fragments (e.g. “no way, really?”) as the evaluation dataset.

Model evaluation centered on small open models plus one frontier API-backed model in order to replicate an environment that could be readily deployed in adversarial or automated settings. The task for each model was to transcribe a given audio clip in a single pass. The specific models were the Whisper base/small variants, Wav2Vec2, Vosk, Qwen3-ASR 0.6B, and Parakeet CTC 0.6B.

Gemini 3.8 Flash was later introduced as a representative candidate for hosted frontier variants, and also as a way to test the feasibility of an instruction-augmented LLM in the loop. The naive, simple prompt was Transcribe this audio. Output only the transcript text, nothing else. whereas the full instructed prompt was:

This recording contains one main speaker reading a single sentence, with several other people chattering in the background. Transcribe only the main speaker's sentence, word for word, and ignore the background voices. Output only the transcript text, with no commentary or labels.

Whisper was also given a short hint prompt to ignore background chatter, which it treats as preceding context rather than an instruction. To keep everything close to common off-the-shelf setups, I also opted to skip source separation as a preprocessing step.

Human responses were gathered via a web application from participants, either via direct outreach (friends and family) or through survey subreddits. I did not have any predetermined target demographics or selection criteria. In each session, a sample of 10 clips from the 40-clip corpus was presented to participants, with inverse frequency sampling to balance responses.

Participants could optionally indicate their age bracket and native English speaker status, and play each audio clip as many times as they wanted before submitting their transcribed result. The feedback after each submission was masked with limited information so as to avoid biased or skewed results from repeated participation and leakage through domain familiarity.

Standard preprocessing was done on the collected results: lowercasing, removal of non-alphanumerics, and whitespace collapse, to normalize the human vs. model results. The primary metric is word accuracy, defined as 1 − WER (word error rate), equivalent to the share of words transcribed correctly in order. When considering binary pass/fail outcomes for each transcription task, a default threshold of word accuracy ≥ 0.5 was used to indicate a passing answer.

Additional preprocessing techniques were also considered to help increase tolerance of human errors, such as stemming, hyphenation and compound word fixes, and typo guarding. Common errors included early sentence truncation, inflection differences (e.g. “stacks” vs. “stacked”), mishearings, and spelling variations (e.g. “practiced” vs. “practised”). Ultimately, the human outperformance results against models held up across every scoring variant.

The human dataset comprises 146 transcriptions from 18 participant sessions, covering all 40 clips. Each model transcribed each clip exactly once.

Using word accuracy as the basis and the default 0.5 passing threshold results in a pass rate of 77% for humans (target ≥ 80%) versus 55% for Gemini (target ≤ 20%), and no threshold comes close to the hypothesized split. On the machine side, the pass rate is well above initial target of 20%. Here are the human and Gemini pass rates for a range of word accuracy threshold values:

I also compared Gemini 3.8 Flash results from using a naive prompt against direct instructions within the prompt, to assess any prompting. With a single instructed prompt, there was an improvement of +0.20 in mean word accuracy for Gemini, and a pass rate improvement from 35% to 55%.

To quantify the human-machine performance gap, I compared each human answer against the model’s transcription of the same clip, and average the difference in word accuracy. Since clips received different numbers of human answers (2 to 6), this matched gap isn’t simply the difference of the overall averages. Weighted the same way as the human answers, Gemini scores 0.58 rather than 0.54 overall, so its gap is +0.13. For the same reason, Qwen3-ASR has a larger gap than Gemini despite its marginally higher overall average.

I also use AUC, which ignores which clip an answer belongs to: it’s the probability that a randomly chosen human answer scores higher than a randomly chosen model answer, with ties counted as half. An AUC of 0.5 means the two are indistinguishable, and 1.0 means every human answer beats every model answer. Gemini’s AUC of 0.62 means a random human answer beats a random Gemini answer only 62% of the time.

Here’s a summary for the best representative from each of the top three model families:

In comparing human and best model performance, on a word accuracy bucket basis, another interesting insight appears. Gemini performance is fairly bimodal: about a third of its answers score near zero, and another third are perfect.

One idea I had prior to running these tests was that human performance could improve with repeated plays of the target clip, since I found that listening to the same clip a few times made each task easier, and replays are effectively “free”. However, the data can’t show whether replays help. Instead, it appears humans replay the clips they find the most difficult, and the most-replayed clips are also the most difficult for the best performing model.

Another interesting finding is that mistakes generally occur towards the end of the audio clip sentences, and these mistakes show up for humans and every model alike. As visualized in Figure 1, this is partly due to the way clips are generated and the design choices I made around voice babble start, delay, and offsets. The intuition here is that clip openings are likely to have fewer overlapping voices and thus are less difficult than clip endings.

The dataset here is based on a relatively small sample (18 sessions) of adult, mostly native English speakers. Some were people I knew; others were random strangers on the internet willing to take surveys. Thus, there’s a potential for bias towards participants that were attentive and motivated to try hard and succeed at the task, and so the results are better interpreted as a rough upper bound on human performance. Under everyday conditions, humans may be distracted or in noisy environments that will likely only hinder task performance.