OpenAI still has a WebRTC Problem

On a previous episode, I flamed OpenAI for using WebRTC and it went viral on Hacker News.

A huge shoutout to Sean DuBois and Justin Uberti (WebRTC experts at OpenAI) for reaching out after my unhinged rant. You could have just said @kixelated ur dumb but you didn’t.

Alas, my divinely anointed crusade against WebRTC is not over. deus vult

Let’s say you’re a vibe coder at a frontier lab working on a voice AI product.

Let’s say you’re making (and burning) an obscene amount of cash.

Let’s say one of your users vocalizes: what is le capital of France?

I’m sorry, you broke up, can you repeat that?

You fucked up. You picked WebRTC for voice AI.

You boot up your favorite application: ChatGPT™. You click on the microphone and grunt:

OH NO, hypothetical sudden packet loss. A kraken nibbles on some underwater cable. All too common I’m afraid.

WebRTC is designed for conference calls, aka real humans and real voices. It is hyper aggressive and will drop any late audio packets to keep latency real-time. Just like Leonardo DiCaprio, anything above 30ms is too old.

NETWhatislecapitalofFrance?WhatislecapitalofFrance?WhatislecapitalofFrance?WhatislecapitalofFrance?“France is a country in Western Europe.”

▶spoken / understoodreceived, not yet understoodlost forever

What is le France?

This kind of sucks for a voice AI product. You walk out of wifi range, hit a cellular dead zone, or starlink terminal switches satellites. Oops, your tokens are wasted, or worse, AI tries to guess what you said.

WebRTC can usually detect packet loss and retransmit. But retransmissions are disabled by default for audio. And even when enabled, the audio jitter buffer won’t wait for them.

But why not? Isn’t this networking 101? Just send that shit again?

We flesh bags consume/produce audio in “real-time”, aka a human clock rate. Our neurons can only fire so fast.

For a real-time conferencing product like Google Meet, latency needs to be near zero. Any delay is super fucking annoying, which is why WebRTC does everything in its power to minimize it.

Blocking on retransmissions will cause delays. It’s the fatal flaw with TCP and why WebRTC is UDP based in the first place. And worse yet, retransmissions will cause a rapid burst of audio on recovery.

This is a bad user experience. Humans do not want to wait, then process a burst of audio at 10x speed. WebRTC instead primarily* relies on human retransmissions.

You: I said, what is le capital of France?

• Before a WebRTC warrior flames me on LinkedIn, read the fix it fix it fix it section.

Of course not. Wake up old man, it’s not 2020 any longer. The internet is no longer for our flabby lips.

Our robot overlords produce/consume audio based on a silicon clock rate. We shoved electricity into a rock and now it can read a library in minutes. What a time to be alive.

Voice agents can hear and speak audio faster than real-time.

Listening at 20x the speed? No problem.

Speaking at 20x the speed? No problem.

Humans stuck at real-time.

We speak too slowly, so speech-to-text has to wait for us.

We hear too slowly, so text-to-speech has to wait for us.

WebRTC is designed for humans, not voice AI agents. These poor robots are being held back by our mediocrity.

When it comes to LLMs, latency budgets get weird.

Normally you would think of latency as the amount of time it takes for a grunt to traverse a tin can. However, LLMs are blocked on context.

Here’s a dumb example of context growing over time:

If an LLM tries to answer my prompt before getting the full context, shit will suck. France is le gatekeeper, and everything before it is effectively a prerequisite. In this example, what matters is the latency of the last packet.

Those poor packets that were attacken by the kraken? WebRTC actually had seconds to retransmit with no latency penalty. The AI agent can process faster than real-time to catch up without the end user (aka YOU) noticing a thing.

So that’s what Mr. MoQ does. No packet left behind: we keep retransmitting until the AI actually needs the audio.

NETWhatislecapitalofFrance?WhatislecapitalofFrance?WhatislecapitalofFrance?WhatislecapitalofFrance?“Paris.”

▶spoken / understoodreceived, not yet understoodwaiting for retransmit

get stuffed mr kraken

It’s not expensive either: audio is tiny. 1s of Opus @ 64Kbps is 8KB, originally sent as 50 tiny packets, but it can be retransmitted as ~6 QUIC packets. We have literal seconds to retransmit 6 packets to avoid any user impact.

And if we manage to accomplish this herculean task: all the context arrives in time!

The model no longer has to guess. It no longer has to be trained on silence or distortion either. This is a pure user experience win.

Guess what? WebRTC doesn’t just suck for STT, it also sucks for TTS.

Fun Fact: WebRTC renders audio (and video) based on when it arrives, not when it is supposed to be rendered. Every other protocol uses timestamps, but Mr. WebRTC barely looks at them, because accuracy is a non-goal. Sure, it’ll nudge audio and video into sync by a few milliseconds, but good luck sending the video 500ms after the audio.

This is a huge problem for voice AI. TTS is generated faster than real-time, stamping each packet with a future timestamp. But in order to play back audio at the correct time with WebRTC… the agent needs to sleep before transmitting each packet.

And if there’s any packet loss, our TTS is lost, will never be retransmitted, tokens burned, yada yada.

NETParis.Homeoflecroissant.Paris.Homeoflecroissant.Paris.Homeoflecroissant.Paris.Homeoflecroissant.“Sorry, you cut out. Home of what?”

▶sent / playedgenerated / bufferedlost forever

When the kraken attacken, there’s no coming backen.

If only there was some way to signal when to render each audio packet. A stamp of sorts.

Welcome back to the future Marty. We faster than real-time now.

RAM has this amazing property of being stupidly expensive. And being able to store stuff locally. You know, like audio that needs to be played over the speakers.

We put a timestamp on each audio frame and use that to determine when to play it out of RAM. Absurd, I know.

NETParis.Homeoflecroissant.Paris.Homeoflecroissant.Paris.Homeoflecroissant.Paris.Homeoflecroissant.“Huh, I didn’t know that.”

▶sent / playedgenerated / bufferedwaiting for retransmit

I’m fast as fuck boye

It’s how the Youtubes keeps playing back video even when you lose network connectivity. MoQ does the same thing but for voice AI. Your sick baking recipe won’t be subject to surprise congestion.

And don’t you fret, 2 minutes of audio is only 1MB. You’re not going to have to take out a loan to buy another RAM stick.

Disclaimer: this behavior is not IETF standardized. It’s something only my moq.dev clients (and thus Pipecat) support. Be my guest if you want to argue with Cisco and get it into the official standards.

WebRTC is the state of the art for real-time, conversational audio. It’s available in every browser via libwebrtc, a C++ monolith maintained by Google. It’s so complicated and sophisticated that Safari and Firefox use libwebrtc too, rather than vibe code their own like everything else.

The problem is that WebRTC is the Google Meet client. It is designed specifically for Google’s conferencing product, not their voice AI product (they use WebSocket), and certainly not your product.

quite the anti-competitive ruse tbh. Google can ship product updates without consulting the other browser vendors. No other conferencing app can do this on the web.

Fun Fact: This is why every other conferencing app recommends a native client. Otherwise, they can only reach parity with Google Meet at best. Native apps can do whatever they want over UDP instead.

So that’s why your hyper deluxe ChatGPT™ voice AI model, after billions invested, has the audacity to ask: can you repeat that?

I could dump on WebRTC for days, but I want to be positive for a change. Here’s how to fix it.

First off, I need to acknowledge what WebRTC does have:

To fix WebRTC, we need to petition Google to:

And of course, petition the wider WebRTC ecosystem to do the same.

The main problem with WebRTC will remain: Google is the gatekeeper.

Your Claude friend can’t just fork libwebrtc and fix these issues. Google is required for web support, at least until Theo spends $420k to rewrite Chromium in Rust.

I think it’s amazing that playout-delay was merged 10 years ago and audio still doesn’t have it. Higher latency/quality playback has never been a priority for WebRTC, since Google Meet is the only product that matters.

Fun Fact: This is why I ended up creating MoQ. We tried using WebRTC at Twitch but the real-time-or-else made for a poor user experience. I am but a humble software engineer, and despite trying, we failed to petition Google.

But OpenAI has enough sway to fix WebRTC. One long-deserved improvement is WARP (not that WARP), which was hot on the heels of my last blog post. Keep at it Justin, or join the Dark Side with me.

In my humble and handsome opinion, it’s time to invest in a better transport. Take a few of the M’s from the B’s and T’s; nobody will notice.

A voice agent should never ask the user to retransmit a prompt.

A voice agent should instead ask the transport to retransmit a prompt.

Spoiler alert: Media over QUIC is that better transport btw.

Pipecat now supports MoQ! A huge thank you to Vanessa and Claude Code for putting in the work and putting up with my constant “improvements”.

Here’s a crude demo with my credit card deets attached. please don’t milk me dry.

Note that I’ve only spent like a day of work prompting on this. It took me significantly longer to write this damn blog post. Don’t judge too harshly pls.

And yes, it sometimes says “can you repeat that?” which is hilariously ironic. The first/last hop is MoQ, but I don’t control the rest of the pipeline. If Deepgram, OpenAI, or Cartesia drop ur packets then there’s nothing I can do about it.

We’re not done yet. I wanted to shill the MoQ in the middle of the blog and use the rest…

Sometimes the AI is monologuing and you need to tell it to STFU. It’s pretty easy to do with WebRTC because of the sleep before sending each packet. But with MoQ, we need some signal to tell the player to abandon the playout buffer.

Here’s the naive approach without a signal:

NETParis.Homeoflecroissant.Whatnext?Paris.Homeoflecroissant.Whatnext?Paris.Homeoflecroissant.Whatnext?Paris.Homeoflecroissant.Whatnext?“Stop!”“I said STOP.”

▶sent / playedgenerated / buffered

I definitely did not do this on my first implementation.

My solution for this is deviously clever. So clever.

For MoQ, we’ve prioritized the ability to add/remove tracks dynamically. Most media protocols have fixed tracks and it’s a pain in the butt: HLS, DASH, RTMP, WebRTC, etc. You can’t add 1080p, or switch codecs, after a broadcast is live.

My super clever idea is to reuse this functionality for interruptions. Just make a new audio track (and remove the old one) as the STFU Signal™.

NETParis.Homeoflecroissant.Paris.Homeoflecroissant.Okay,whatshouldwediscussnext?Okay,whatshouldwediscussnext?Paris.Homeoflecroissant.Paris.Homeoflecroissant.Okay,whatshouldwediscussnext?Okay,whatshouldwediscussnext?“Stop!”“Croissants.”

▶sent / playedgenerated / buffereddiscarded

home of le crepes

This works on a generic MoQ player. We automatically switch tracks, using timestamps to keep synchronization.

Hey stupid, your blog post is wrong. You didn’t mention speech-to-speech?

Look, Dunning-Kruger is the only reason the previous blog post exists. I didn’t know speech-to-speech existed until the comment section. What can I say but ur welcome.

DISCLAIMER: I still haven’t done any research on the state of voice AI. I give you full permission to tweet @kixelated ur stupid.

The idea behind speech-to-speech is to cut out the text middleman. The disjoint STT -> LLM -> TTS pipelines get smashed together, reducing latency while preserving verbal context. OpenAI