
Cohere Transcribe hits 1.46% WER clean, 2.5x worse than claimed on real audio
All articlesCoherespeech-to-textASRtranscriptionWERH100benchmarkvLLMRTFxLibriSpeechAMI
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
All articlesCoherespeech-to-textASRtranscriptionWERH100benchmarkvLLMRTFxLibriSpeechAMI
Cohere Transcribe (03-2026) on H100: 1.46% WER on clean speech, but the published 5.42% claim doesn't survive contact with real audio
We ran CohereLabs/cohere-transcribe-03-2026 across LibriSpeech-clean, AMI meetings, and VoxPopuli, then duration sweeps from 1 to 8 H100s. Strong on the easy set (1.46% WER), 2.5× worse than Cohere's published numbers on harder real-world audio, and the multilingual evaluation broke completely. Here's what we measured and why it matters for buyers.
Copy linkShare on LinkedIn
On this page
TL;DRWho this is forWhat Cohere Transcribe isFor the buyer: stack against Whisper, hosted APIs, and the published claimCost reality at scaleFor the operator: throughput, RTFx, and GPU scalingFor the researcher: where the numbers don't line upThe AMI WER gapMultilingual evaluation: 0 working languagesRobustness: no hallucinations on non-speechFor the developer: how to actually run itPatterns the run confirmedBottom line
TL;DR
Cohere released cohere-transcribe-03-2026 in March 2026 — a speech-to-text model published alongside aggressive accuracy claims (5.42% average WER across a four-dataset eval). We ran the model on H100 hardware, across 1, 2, 4, and 8-GPU configurations, with the same four datasets they cited plus a stress sweep on duration and batch size.
Cohere Transcribe 03-2026 is a speech-to-text model. The architecture details Cohere has disclosed are thin — they published it as a hosted API and as a downloadable checkpoint without a detailed tech report. From the model card and our profiling, it's clearly an encoder-decoder transformer trained on a large multilingual corpus, sized to be deployable on a single GPU for production transcription workloads.
The model is published under Cohere's commercial license and is positioned as a direct competitor to OpenAI Whisper Large v3 and the major hosted ASR APIs (Deepgram, AssemblyAI, Speechmatics). Cohere's marketing claims a 5.42% average WER across LibriSpeech-clean, LibriSpeech-other, AMI-IHM, and VoxPopuli, which would put it above Whisper Large v3 on the same datasets.
Three commercial alternatives to evaluate against:
Three numbers that decide the buy/build evaluation:
The accuracy comparison is the chart that matters. On LibriSpeech-clean we measured 1.46% WER — basically at parity with the public ASR leaderboard's best results and consistent with Cohere's claim. On VoxPopuli-EN we measured 6.25% WER — credible for noisy parliamentary speech. But on the two AMI meeting datasets — close-talk individual headset microphones (AMI-IHM) and single distant microphone (AMI-SDM) — we measured 17.74% and 29.17% WER respectively, dragging the average up to 13.65%.
Cohere's published number is 5.42%. The gap is large enough that it can't be attributed to test-set variance. There are three plausible explanations:
Until that gap is resolved, the buyer's decision is conditional: for clean studio-quality audio and clean podcast speech, Cohere Transcribe is excellent. For meeting recordings, conference calls, and far-field microphone audio, expect WER above 25%, and budget for a fallback or a domain-specific fine-tune.
A self-hosted single H100 ($2.50/hour on Lambda Labs) processes audio at ~12× real-time. That's $0.21 per hour of audio — cheaper than every hosted alternative and competitive with self-hosted Whisper Large v3. Above ~2,000 hours of audio per month, this break-even is decisive. Below it, the hosted-API path wins on ops simplicity.
We ran duration sweeps from 10-second to 5-minute clips across 1, 2, 4, and 8 H100 configurations. The key finding is that single-GPU performance is the operationally relevant number — multi-GPU scaling adds throughput but not RTFx per request.
The Pareto frontier across batch sizes for different audio clip lengths:
Three gaps between what we measured and what Cohere published. Each is worth reproducing before drawing conclusions.
On AMI-IHM (close-talk individual headset mic) we measured 17.74% WER. On AMI-SDM (single distant mic) we measured 29.17% WER. Cohere's published number for AMI is in the 6-8% range. The gap is large.
Looking at the per-sample errors, the model handles short utterances well but degrades sharply on overlapping speech, low-SNR clips, and clips with mid-sentence speaker changes. AMI's meeting recordings are dominated by exactly those conditions. Cohere may be reporting on a cleaner AMI sub-split, may use a longer beam-search width than vLLM's default, or may pre-process with VAD chunking that we didn't apply. All three are testable.
The multilingual benchmark attempted 14 languages × 30 samples each. All 14 returned a "no_samples" error. The model didn't fail to transcribe — the test harness failed to load any audio samples. This is a benchmark-infrastructure bug, not a model bug, but it means we have no data on Cohere Transcribe's multilingual quality.
The fix is straightforward: the multilingual evaluation expects samples in a specific HuggingFace dataset format that wasn't downloaded; re-running with the correct dataset path is on the to-do list. Until that's done, treat the multilingual claims as unverified by us.
On the robustness suite (silence, music, white noise, phone codec), the model returned empty transcriptions or minimal hedged outputs — no hallucinated speech content. That's the right behavior. Whisper and earlier ASR models are notorious for confabulating coherent transcripts on silence; Cohere Transcribe doesn't.
Under degraded signal conditions (phone-codec 8 kHz, low SNR), WER scales gracefully rather than collapsing. The chart above shows the SNR curve — at -5 dB SNR the model is still producing useful transcriptions, at -10 dB it starts producing partial output, at -15 dB it gives up cleanly. Better than the silent-failure modes of older models.
uv venv .venv --python 3.12 source .venv/bin/activate uv pip install vllm==0.22.1 librosa soundfile huggingface_hub
serve on 1 H100 (multi-GPU offers throughput, not lower latency)
vllm serve CohereLabs/cohere-transcribe-03-2026
--tensor-parallel-size 1
--max-num-seqs 8
First transcription call:
import openai, base64 client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
audio_b64 = base64.b64encode(open("clip.wav", "rb").read()).decode() resp = client.audio.transcriptions.create( model="CohereLabs/cohere-transcribe-03-2026", file=("clip.wav", base64.b64decode(audio_b64)), ) print(resp.text)
Three gotchas:
Cohere Transcribe 03-2026 is a credible Whisper Large v3 alternative for clean speech use cases — podcast transcription, dictation, studio recordings, structured-content audio. The cost story at scale is compelling: ~$0.21 per hour of audio on self-hosted single H100, well below hosted alternatives, above ~2,000 hours/month of audio.
But the model is not yet validated for meeting / far-field / multilingual use cases. The accuracy gap on AMI is too large to ignore, and the multilingual evaluation infrastructure failed completely. Until those gaps close, treat the published 5.42% average WER as aspirational rather than verified.
We'll re-run with VAD chunking, the correct multilingual sub-split, and a head-to-head against Whisper Large v3 on the same audio. That's a follow-up post. Reproduction script is at cohere-transcribe-03-2026/run_all.sh in the internal ML-Models repo.
LocateAnything-3B benchmarked on a 12 GB GPU across 10 dense production scenes: traffic, PPE, cattle, warehouse, retail. Peak VRAM 9.7–11.1 GB, latency 7.4–19 s per scene, with one revealing failure mode. Real numbers, real images, honest write-up.
Llama 4 Maverick benchmarked on 8×H100 across 10 evaluations: 3,714 tok/s throughput, MT-Bench 9.03, 100% on coding/vision/multilingual/long-context, 90% on IFBench, and 0% on tool-use — read why before you ship.
100% coding accuracy across 8 categories, 9.57 MT-Bench, 93% tool use, 8,407 tok/s. Our deployment evaluation for engineering teams considering self-hosted code AI.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。