返回资讯中心
外部精选
软件工程
#开发者实时流

Measuring voice agents: turn-taking, tool calls and expressive speech

October 30, 2026 · IvanUpdated October 31, 2026

Hacker News 最新aufklarer2026/10/31 · 17 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

BlogBenchmarks

October 30, 2026 · IvanUpdated October 31, 2026

Measuring voice agents: turn-taking, tool calls and expressive speech

A voice agent has to choose when to speak, which action to take, and how its reply should sound. We measured those decisions separately, using local models and cloud API configurations. The results help us choose components for private voice applications.

In our earlier survey of full-duplex speech and tools, we followed a spoken request through listening, interruption, tool execution and synthesis. This post adds measurements. You can inspect the complete leaderboard and the saved snapshot behind these tables.

Turn ends caught, false alarms and decision latency.

Correct action selection and argument accuracy.

Following a voice description, delivery direction or role.

We measure these three abilities separately. An assistant can wait for you to finish speaking and still choose the wrong tool. It can also speak clearly while missing the emotion or style you asked for.

We ran the local models on an Apple M5 Pro with 48 GiB of memory. For cloud services, we tested the model versions and settings listed in the results.

Each section explains how many recordings we used, which languages we tested and any results still being checked. These details help you compare the scores.

TurnBench distinguishes a speaker finishing from a pause inside a turn. It also distinguishes an interruption from a backchannel such as “mm-hmm.” Our table below focuses on end-of-turn detection.

We used TurnBench’s public development recordings and its own scoring code. The table shows the run size for each model. The official leaderboard uses a separate test set.

Recall is the share of real turn endings the model detects. FPR (false-positive rate) is the share of labelled pauses within a turn that it incorrectly treats as the speaker finishing.

Our leaderboard orders models by recall minus FPR. The official test leaderboard ranks models by recall, placing those with an FPR above 15% below those that meet the limit.

We added four TurnBench submissions by rescoring their published predictions on the same five conversations, without changing their decisions or rerunning their models. Nunchi v2 has the highest recall-minus-FPR score on this slice: 96.6% recall and 6.4% FPR. VAP reaches 88.2% recall with 7.4% FPR; Mimi makes fewer false detections, at 2.0%, but catches fewer turn endings and takes longer to decide.

VAP (Voice Activity Projection) listens to both speakers on separate audio channels and predicts their upcoming speech activity. It uses the recent conversation to estimate whether the current speaker will continue or hand over to the other person. Research on VAP shows that it learns to use cues in the voice, such as pitch and rhythm.

The version we ran was trained further on 104 hours of conversational audio from otoSpeech. Our interpretation is that this training and its view of both speakers help explain its strong result in our comparison.

We tested three downloadable exports on two TurnBench development conversations: 148 turn endings and 94 pauses inside a turn. Each run used a fixed decision policy.

At these settings, p99 caught ten more turn endings, while VAP made thirteen fewer false detections. We got the same event decisions from each export and its original checkpoint on the same audio. The pilot protocol records the inputs and cutoffs.

Both models now have speech-swift APIs. We also added VAP to the C++ and Android runtimes. The integration guide covers downloads and audio inputs. Your application sets the response policy.

SLURP contains spoken home-assistant requests with an annotated action and argument values. In our routing test, each recording comes with ten candidate actions and a no-action option.

Our 1,000-recording setup includes 810 requests with the correct action available and 190 with that action withheld. We expect no action on those 190 requests. This checks whether the model declines a request when its action is unavailable.

Most systems use 1,000 recordings, including close-talk and far-field audio. Far-mic accuracy measures correct choices on recordings made from farther away, when the requested tool is available. Calls despite a missing tool counts requests where the system returned an action even though the annotated tool was absent. It pools close-talk and far-field recordings, and does not measure activation from background speech or silence. The table shows the exact counts.

The local Parakeet → Gemma 4 E2B pipeline reaches 75.2% far-mic accuracy, close to Nova’s 75.9% and behind GPT Audio’s 83.0%. But on the 190 requests whose correct tool was unavailable, Gemma still chose an action 88 times: its 46.3% false-activation rate. One example was “delete item”: with list removal unavailable, it chose calendar removal. It also rejected 88 requests whose correct tool was available and selected the wrong tool on another 69.

Jev and GLiNER both choose the correct tool on 311 of the 423 supported far-mic requests, giving 73.5% accuracy. Their behaviour changes when the correct tool is missing: Jev still chooses an action on 22 of 190 requests (11.6%), while GLiNER does so on 74 (39.0%). For example, GLiNER interpreted a request to lower speaker volume as dimming a light when the volume tool was absent. These examples selected a different action from the one requested. Broader tool descriptions can also overlap: a recipe request may be sent to a general cooking tool. Our count follows the annotated intent, so it does not establish that every alternative was semantically unreasonable.

Nova has the highest score for matching both the tool and all required argument values: 45.4%, or 368 of 810 supported requests. It chose the correct tool on 643 requests, but the arguments failed the match on 275 of those. In 147 cases the values differed from the reference; another 128 omitted at least one argument, sometimes alongside value differences.

The most frequent value mismatch was a time, appearing in 34 of those cases. We also saw missing dates and misnamed fields. For an alarm request that included “this week,” Nova returned the time but left out the date. Some failures are wording differences: “lasagna” does not match the reference’s “lasagne” under this strict text comparison. The score counts both kinds of mismatch.

For InstructTTSEval, we keep the spoken script fixed and generate separate clips using three types of instruction:

The percentage for each type shows how often the generated speech followed that instruction.

The larger cohort contains 25 English and 25 Chinese source lines, three checks each: 150 clips per system. Nova covers the 25 English lines, with 75 checks. Overall is total passes divided by total checks; with equal task sizes, it is the mean of APS, DSD and RP. A saved Gemini Pro judge supplies the style verdicts, while transcription checks the spoken words separately.

Original recordings are the benchmark’s human source audio. We score them with the same instructions and judge used for the 150-check runs to give the synthesized voices a reference point. Their 73.3% means the judge accepted 110 of 150 instruction checks.

5 English + 5 Chinese lines; 30 style checks

25 English + 25 Chinese lines; 150 style checks

25 English + 25 Chinese lines; 150 style checks

5 English + 5 Chinese lines; 30 style checks

25 English + 25 Chinese lines; 150 style checks

Provisional · 1 judge response under review

25 English + 25 Chinese lines; 150 style checks

25 English + 25 Chinese lines; 150 style checks

Provisional · 1 judge response under review

25 English + 25 Chinese lines; 150 style checks

Provisional · 1 judge response under review

25 English + 25 Chinese lines; 150 style checks

25 English + 25 Chinese lines; 150 style checks

Experimental input · 1 judge response under review

25 English + 25 Chinese lines; 150 style checks

Input conditioning and judge response under review

25 English + 25 Chinese lines; 150 style checks

Provisional · 2 judge responses under review

On the matched 150-check cohort, FireRedTTS3-Instruct scores 65.3% and MOSS-VoiceGenerator scores 56.0%. Google’s TTS run scores 67.3%. On the 75 English checks, FireRed scores 72.0%, MOSS 52.0% and Google 72.0%. VoxCPM2’s BF16 run remains at 59.3%; its role-play result, 46.0%, is weaker than its voice-description and direction results. The larger Qwen CustomVoice checkpoint scores 52.7%, versus 48.0% for the smaller experimental-input run. Size alone did not determine the score.

Our ElevenLabs instruction-to-tag conversion dropped requested singing and some emotion changes. CosyVoice3 failed the word-content check on 48 of 50 APS clips; its generation setup also omitted reference conditioning present in the official implementation. These issues still need corrected-input validation. Fish’s corrected run and several other rows contain judge responses that claimed audio was unavailable; we preserve those verdicts and mark the affected rows provisional. These are results for our ports, input adapters and judge, rather than a finding that an upstream model has the same limits in every implementation.

Most failures on the human recordings are in role-play: 29 of 50 checks fail, compared with eight for APS and three for DSD. A real recording can fit a detailed voice description yet not match the role assigned to it, and the judge can also make mistakes. The reference row helps us interpret instruction scores; it does not rate the quality of human speech.

Qwen’s VoiceDesign checkpoint generates a voice from a written description. CustomVoice instead applies directions to a selected preset voice. The checkpoint and control channel matter: a field named “instruction” does not give a model a capability it was never trained to use.

We tested Qwen3-TTS 1.7B VoiceDesign and Breeze TTS 2 on the same five English and five Chinese sources, giving 30 checks per model. Four calibration checks are outside their scores. We retained all outputs, including one spoken-word flag per model.

VoiceDesign passed 23/30 checks and Breeze passed 20/30. With ten source recordings shared across the three tasks, this is a development screen. The paired 10-point difference has a wide source-cluster interval that includes zero. The judge setting also differs from the larger cohort, so the pilot does not establish that VoiceDesign outperforms Google’s 67.3% result.

A voice-design request, not a benchmark result

Spoken text: “Your delivery is scheduled for Friday morning.”

Voice brief: “An adult British English narrator with a warm tone, clear articulation and an unhurried pace.”

Keep the script unchanged when trying another delivery. A designed voice and a clone from an authorised reference are different inputs; test consistency across the whole passage.

FireRedTTS3-Instruct and MOSS-VoiceGenerator now each have 150 style checks in the table above. The separate content check flagged 0/150 FireRed clips and 12/150 MOSS clips for word error above 30%. Their short-script runtime samples remain in the Space. First available PCM was about 4.50 s for FireRed and 0.47 s for MOSS on the warm English script; FireRed returns a complete waveform while MOSS streams. Those timings do not establish instruction quality or a matched quality-speed ranking. Existing-model latency and memory certification remains inconclusive.

Research snapshot: 30 October 2026. The official TurnBench held-out board includes newer submissions beyond the paper’s original baselines. p99’s system write-up describes 6.1 million learned parameters, extra classification heads and a turn policy that uses both audio channels. Its Apache 2.0 download contains the 5.3-million-parameter mono model. We measured that downloadable checkpoint in the export pilot above.

LiveKit’s eot-bench complements this with real human-to-agent turns in 14 languages. It sweeps endpointing policies and compares lat

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →