Raw capture ā raw/webrtc-is-the-problem.md
url: https://moq.dev/blog/webrtc-is-the-problem/ date_fetched: 2026-07-05 backfilled: true
published 5/5/2026
OpenAI posted a technical blog a few days ago. This blog post triggered me more than it should have. I urge to slap my meaty fingers on the keyboard.
You should NOT copy OpenAI.
I donāt think you should use WebRTC for voice AI. WebRTC is the problem.
Like 6 years ago I wrote a WebRTC SFU at Twitch. Originally we used Pion (Go) just like OpenAI, but forked after benchmarking revealed that it was too slow. I ended up rewriting every protocol, because of course I did!
Just a year ago, I was at Discord and I rewrote the WebRTC SFU in Rust. Because of course I did! Youāre probably noticing a trend.
Fun Fact: WebRTC consists of ~45 RFCs dating back to the early 2000s. And some de-facto standards that are technically drafts (ex. TWCC, REMB). Not a fun fact when you have to implement them all.
You should consider me a Certified WebRTC Expert. Which is why I never, never want to use WebRTC again.
Iām going to cheat a little bit and start with the hot takes before they get cold. Donāt worry, weāll get right back to talking about the OpenAI blog post and load balancing, I promise.
WebRTC is a poor fit for Voice AI.
But that seems counter-intuitive? WebRTC is for conferencing, and that involves speaking? And robots can speak, right?
Letās say I pull up my OpenAI app on my phone. I say hi to ~~Scarlett Johansson~~ Sky and then I utter:
should I walk or drive to the car wash?
WebRTC is designed to degrade and drop my prompt during poor network conditions.
wtf my dude
WebRTC aggressively drops audio packets to keep latency low. If youāve ever heard distorted audio on a conference call, thatās WebRTC baybee. The idea is that conference calls depend on rapid back-and-forth, so pausing to wait for audio is unacceptable.
ā¦but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate. After all, Iām paying good money to boil the ocean, and a garbage prompt means a garbage response. Itās not like LLMs are particularly responsive anyway.
But Iām not allowed to wait. Itās impossible to even retransmit a WebRTC audio packet within a browser; we tried at Discord. The implementation is hard-coded for real-time latency or else.
UPDATE: Some WebRTC folks are claiming this is a skill issue. It might be possible to enable audio NACKs, but we couldnāt figure out the correct SDP munging. Either way, the WebRTC jitter buffer is aggressively small.
And yes, Voice AI agents will eventually get the latency down to the conversational range. But reducing latency has trade-offs. Iām not even sure that purposely degrading audio prompts will ever be worth it.
You speak into the microphone, it gets sent to one of OpenAIās billion servers, and then a GPU pretends to talk to you via text-to-speech. Neato.
Letās say it takes 2s of GPUs to generate 8s of audio. In an ideal world, we would stream the audio as itās being generated (over 2s) and the client would start playing it back (over 8s). That way, if thereās a network blip, some audio is buffered locally. The user might not even notice the network blip.
But nope, WebRTC has no buffering and renders based on arrival time. Like seriously, timestamps are just suggestions. Itās even more annoying when video enters the picture.
To compensate for this, OpenAI has to make sure packets arrive exactly when they should be rendered. They need to add a sleep in front of every audio packet before sending it. But if thereās network congestion, oops we lost that audio packet and itāll never be retransmitted.
OpenAI is literally introducing artificial latency, and then aggressively dropping packets to ākeep latency lowā. Itās the equivalent of screen sharing a YouTube video instead of buffering it. The quality will be degraded.
Fun fact: WebRTC actually adds latency. Itās not much, but WebRTC has a dynamic jitter buffer that can be sized anywhere from 20ms to 200ms (for audio). This is meant to smooth out network jitter, but none of this is needed if you transfer faster than real-time.
Okay but letās talk about the technical meat of the OpenAI article. Weāre no longer on a boat, but letās talk about ports.
When you host a TCP server, you open a port (ex. 443 for HTTPS) and listen for incoming connections.
The TCP client will randomly select an ephemeral port to use, and the connection is identified by the source/destination IP/ports.
For example, a connection might be identified as 123.45.67.89:54321 -> 192.168.1.2:443.
But thereās a minor problemā¦ client addresses can change. When your phone switches from WiFi to cellular, oops your IP changes. NATs can also arbitrarily change your source IP/port because of course they can.
Whenever this happens, bye bye connection, itās time to dial a new one. And that means an expensive TCP + TLS handshake which takes at least 2-3 RTTs. The users definitely notice the network hiccup when youāre live streaming.
WebRTC tried to solve this issue but made things worse. Seriously.
A WebRTC implementation is supposed to allocate an ephemeral port for each connection. That way, a WebRTC session can identified by the destination IP/port only; the source is irrelevant. If the source IP/port changes, oh hey thatās still Bob because the destination port is the same.
But as OpenAI corroborates, this causes issues at scale becauseā¦
You could probably abuse IPv6 to work around this, but IDK I never tried. Twitch didnāt even support IPv6ā¦
So most services end up ignoring the WebRTC specifications. Because of course they do. We mux multiple connections onto a single port instead.
At Twitch I literally hosted my WebRTC server on UDP:443.
Thatās supposed to be the HTTPS/QUIC port, but lying meant we could get past more firewalls.
Like the Amazon corporate network, which blocked all but ~30 ports.
Discord uses ports 50000-50032, one for each CPU core. As a result it gets blocked on more corporate networks. But like, if youāre on a Discord voice call on the Amazon corporate network, you probably wonāt be there much longer anyway.
HOWEVER, HUGE PROBLEM.
WebRTC is actually a bunch of standards in a trenchcoat, and 5 of those go over UDP directly. Itās not hard to figure out which protocol a packet is using, but we need to figure out how to route each packet.
ufragand route on it.ssrc(u32)ā¦ which we can- usuallyroute based on.So OpenAI only uses STUN:
No protocol termination: Relay parses only STUN headers/ufrag; it uses cached state for subsequent DTLS, RTP, and RTCP, keeping packets opaque.
Itās a positive way of saying:
We really hope the userās source IP/port never changes, because we broke that functionality.
While itās impressive load balancing anything at OpenAI scale, their custom load balancing is a hack. But a necessary hack, because the core protocol is at fault.
Fun fact: Browsers can randomly generate the same ssrc.
If there is a collision, and no source IP/port mapping is available, Discord attempts to decrypt the packet with each possible decryption key.
If the key worked, hey we identified the connection!
The OpenAI blog post starts with 3 requirements, one of them is:
lol
It takes a minimum of 8 round trips (RTT) to establish a WebRTC connection. While we try* to run CDN edge nodes close enough to every user to minimize RTT, it adds up.
Signaling server (ex. WHIP):
Media server:
2 for SCTP
Itās complicated to compute, because some protocols can be pipelined to avoid 0.5 RTT. Kinda like half an A-Press.
All of this nonsense is because WebRTC needs to support P2P. It doesnāt matter if you have a server with a static IP address, you still need to do this dance.
Itās extra depressing when the signaling and media server are running on the same host/process. You end up doing two redundant and expensive handshakes. Itās like walking AND driving your car to the car wash.
Fun Fact: This was originally going to be a Fun Fact, but it gets its own section now.
WebRTC practically encourages you to fork the protocol. Thereās so many limitations that Iāve barely scratched the surface. The browser implementation is owned by Google and tailor made for Google Meet, so itās also an existential threat for conferencing apps.
Sad Fact: Thatās why every conferencing app (except Google Meet) tries to shove a native app down your throat. Itās the only way to avoid using WebRTC.
OpenAI definitely has the ~~debt~~ funding to do this. But I think they should also throw the baby out with the bath water. Donāt fork WebRTC, replace it with something that has browser support.
Fun Fact: Discord has forked WebRTC so hard that native clients only implement a tiny fraction of the protocol. No more SDP/ICE/STUN/TURN/DTLS/SCTP/SRTP/etc. But we still have to implement everything for web clients.
If not WebRTC, then what should you use for Voice AI?
Honestly, if I was working at OpenAI, Iād start by stream audio over WebSockets. You can leverage existing TCP/HTTP infrastructure instead of inventing a custom WebRTC load balancer. It makes for a boring blog post, but itās simple, works with Kubernetes, and SCALES.
I think head-of-line blocking is a desirable user experience, not a liability. But the fated day will come and dropping/prioritizing some packets will be necessary. Then I think OpenAI should copy MoQ and utilize WebTransport, becauseā¦
QUIC FIXES THIS
Remember the round trip discussion? Good times. Hereās how many RTTs it takes to establish a QUIC connection:
But that was an easy one. Letās dive into the deeper details of QUIC that you wouldnāt know about unless youāre a turbo QUIC nerd (it me).
Remember that link to RFC9146? In the DTLS section? That you didnāt click? Good times. The idea is literally copied from QUIC.
QUIC ditches source IP/port based routing.
Instead, every packet contains a CONNECTION_ID, which can be 0-20 bytes long.
And most importantly for us: itās chosen by the receiver.
So our QUIC server generates a unique CONNECTION_ID for each connection.
Now we can use a single port and still figure out when the source IP/port changes.
When it does, QUIC automatically switches to the new address instead of severing the connection like TCP.
But if your gut reaction is: āhow dare they! this is a waste of bytes!ā These bytes are very important, keep reading u nerd.
I glossed over this, but OpenAIās load balancers (like most) depend on shared state. Even if you have a sticky packet router, load balancers can still restart/crash. Something has to store the mapping from source IP/port -> backend server.
Theyāre using a Redis instance to store the mapping of source IP/port to backend server. Simple and easy, I approve.
But do you know what is even simpler and easier? Not having a database. Hereās how QUIC-LB does it:
When a client initiates a QUIC connection, the load balancer forwards the packet to healthy backend server.
The backend server completes the handshake and encodes its own ID into the CONNECTION_ID.
That way every subsequent QUIC packet contains the ID of the backend server.
Now packets become trivial for load balancers to forward. They donāt need encryption keys or a routing table, just decode the first few bytes and forward it to that guy. It doesnāt even matter if the server reboots.
Zero state also means zero global state. These load balancers could listen on a global anycast address and forward packets globally to the indicated backend server. Cloudflare uses this extensively; no need for a global Redis cluster.
Unpaid Shill: AWS NLB offers QUIC load balancing using QUIC-LB. Other cloud providers need to step up their game and offer it too.
Based on the OpenAI blog, it sounds like they assign connections to regional load balancers. Functional but lame. Anycast is way cooler.
I brought this up in my ancient Quic Powers blog post, but Iāll excuse you for not reading it (yet).
QUIC has something called preferred_address that is a game changer for load balancing.
Letās say we have thousands of backend servers around the world that could accept a new connection.
We have them all advertise the same anycast address, ex. 1.2.3.4.
When a client tries to connect to 1.2.3.4, the magic internet routers forward the packet to one of the servers.
Now, we could just use QUIC-LB and route traffic to the indicated backend. But that would be boring.
Instead, we can give each QUIC server a unique unicast address, ex. 5.6.7.8.
The idea is that we use anycast for handshakes and unicast for stateful connections.
1.2.3.4and- 5.6.7.8.1.2.3.4.preferred_address=5.6.7.8.5.6.7.8.When the server is overloaded and doesnāt want more connections, it stops advertising 1.2.3.4.
We wonāt drop existing connections because theyāre safe on unicast.
Just like that, no load balancers needed! The anycast address is basically a health check!
Holy shit I wish I actually had the scale to build this. Reach out if you work for the orange butthole company.
WebRTC
QUIC
I know many engineers at OpenAI and they are extremely bright. Theyāre dealing with unprecedented levels of stress. They MUST scale and they MUST scale now.
Iām just some guy who quit my job to work on a passion project. I literally spend my time tracing memes. Itās easy for me to judge from my lofty position, like a movie critic ranting about how they casted Jared Leto again?
I just donāt think the obvious solution is a good fit for Voice AI. And the obvious solution is very difficult to scale. WebRTC is Jared Leto. There I said it.
And Iāll be honest, MoQ isnāt a perfect fit for Voice AI either. Itāll work, but a lot of the cache/fanout semantics are useless for 1:1 audio. You should definitely use QUIC though.
Anyway, hit me up if you want to chat: meself@kixel.me
Iām cool. You wonāt regret it. Probably.
Written by @kixelated.