setCodecPreferences is now in all browsers!
WebRTC enables video calls directly in browsers. Effectively managing the codecs that encode and decode media streams is a crucial component of delivering high-quality audio and video. The setCodecPreferences method allows developers to specify which codecs their applications prefer to receive.
With Firefox Nightly 128, setCodecPreferences is now available in all browsers! Let’s explore how it functions, its implications for web developers, and its role in harmonizing browser behaviors.
Firefox supporting this WebRTC 1.0 feature is a big deal. It helps webpages work the same across all browsers, eliminating browser-conditional code or having to resort to munging WebRTC’s signaling messages written in SDP, by hand.
But what made it seem blog-worthy was this question: do people know how it works? Looking at MDN and recent spec changes suggested not. Contrary to popular belief, it doesn’t control what you send. At least not directly.
How does setCodecPreferences work?
setCodecPreferences is a method on a WebRTC transceiver. It lets webpages sort and filter the decoders to be considered when negotiating with a peer over a connection. The following illustrates a baseline for usage e.g. on a video transceiver:
transceiver.setCodecPreferences(RTCRtpReceiver.getCapabilities("video").codecs)
This sets the browser’s whole array of available decoders unmodified. Perhaps surprisingly, this is not a no-op. It restricts the browser to consider all its codecs in this specific order. This technically differs from the default transceiver.setCodecPreferences([]) which doesn’t restrict the browser in this way. E.g. a browser may choose to leave out some codecs by default.
It’s also an imperfect API for other reasons. The codecs array contains a mix of real decoders (often repeated with different profile levels) along with special entries for features called RTX, RED and ULPFEC (thanks to JSEP).
As such, it’s not advisable to brute-force your own fixed array e.g. setCodecPreferences([h265, h264]), as this would inadvertently disable RTX, RED and ULPFEC. Instead, we’ll demonstrate best practice: carefully sort the array, using a sortByMimeTypes(codecs, preferredOrder) function below.
setCodecPreferences sets what encodings you prefer to receive
The important part to remember is that setCodecPreferences sets what media encodings you prefer to receive, not what you want to send. Take this example:
If you click the “Result” tab, you should see more or less the following across browsers (use Firefox 128+):
pc1 prefers video/H264,video/VP8,video/VP9,video/ulpfec,video/red,video/rtx pc2 prefers video/VP9,video/VP8,video/H264,video/ulpfec,video/red,video/rtx checking connected pc1 is sending video/VP9 pc2 is sending video/H264
You’ll see pc1 sending VP9 because that’s what pc2 prefers to receive, and pc2 sending H.264 because that’s what pc1 prefers to receive. In other words, setCodecPreferences sets preferences for reception.
Does setCodecPreferences configure a codec for sending?
It does that too sometimes, sort of, implicitly. To understand why, it helps to appreciate how setCodecPreferences is rooted in the offer/answer exchange, where what encoder a sender uses is the result of SDP negotiation with its peer.
The simplest way to see this is to consider what happens when only pc1 expresses a preference:
Now, if you click the “Result” tab, you’ll see pc1‘s preference controlling what both peers send:
pc1 prefers video/H264,video/VP8,video/VP9,video/ulpfec,video/red,video/rtx pc2 has no preference checking connected pc1 is sending video/H264 pc2 is sending video/H264
Essentially, the JSEP offer/answer exchange negotiates what to use with the peer. The devil is in the details, but if only the offerer (pc1) has a preference then it simplifies the negotiation quite a bit.
Especially when the transceiver’s direction is "sendonly", this makes it appear like pc1‘s preference is controlling what it sends, because it is, implicitly (provided the peer has no preference).
Since a transceiver’s direction can be renegotiated over time, having codec negotiation work (mostly) independently of direction is a nice invariant (even if JSEP sometimes gets in the way, more on that below).
Looking at setCodecPreferences as a sender-control leads to surprises
Unfortunately, the pc1 preference trick above to control both sides doesn’t work on pc2:
Click the “Result” tab, you’ll see pc2‘s preference only controls what it receives:
pc1 has no preference pc2 prefers video/VP9,video/VP8,video/H264,video/ulpfec,video/red,video/rtx checking connected pc1 is sending video/VP9 pc2 is sending video/VP8
This is because SDP negotiation is inherently asymmetric: pc1 is the offerer and pc2 is the answerer. By the time pc2’s call to setCodecPreferences is considered the offer has already happened.
This is a case where reversing roles would yield different negotiation results, so it’s good to be aware of it.
But this anomaly is a result of our expectation to control sending. If we shift our mental model to setCodecPreferences controlling reception, then it should hopefully yield fewer surprises.
Live updates and the negotiationneeded event
setCodecPreferences can also be used to make runtime changes mid-stream. Below we see pc1‘s preferences controlling both sides again, except now with a drop-down picker letting us make live changes:
Clicking the “Start!” button, you should be able to use the drop-down and switch codecs live. But behind the scenes, renegotiation needs to happen for these changes to take effect. It’s perhaps surprising that (as of this writing) calling setCodecPreferences() does not trigger the negotiationneeded event! I’ve raised an issue with the spec about this. Hopefully this can be fixed, but that’s going to take some time, so in the meantime the fiddle works around this by negotiating explicitly.
To instead change encoder without negotiation (choosing between already negotiated codecs), look for a future blog post on setting codec with setParameters.
When a browser can decode what it cannot encode
Some concerns were raised on the spec recently about decode-only codecs. For example, Chrome might initially only be able to decode H.265 but not encode it, whereas Safari might both encode and decode it.
It’s a limitation of JSEP that there is no way to indicate per-codec directions in SDP negotiation (where codecs apply to the whole m-line). As a result, some of the spec language, if taken literally, would have browsers exclude decode-only codecs from their sendonly and sendrecv transceivers.
But current thinking based on what Justin Uberti suggested is that this is too strict of a read. Instead, the leading idea is Chrome could offer (to receive) H.265 with H.264 as a fallback. This way Chrome can negotiate successfully with Safari to have Safari send it H.265 while it sends H.264 back.
This seems like a good solution that wouldn’t require any changes, and it aligns well with an understanding of setCodecPreferences as a means to control reception. In short, the offerer includes a fallback codec it can send with. This is another good reason to never truncate the array when using this API. It largely avoids this problem.
Conclusion
Hopefully, this post has helped clarify how setCodecPreferences works. While the method primarily dictates the codecs your application prefers to receive, it also indirectly influences what is sent during peer-to-peer exchanges. This dual role in the negotiation process is helpful to understand for effective use of the API. The method is one of only two on the RTCRtpTransceiver, an interface designed to simplify complex underpinnings. As you integrate this feature into your projects, you’ll likely appreciate both its capabilities and its challenges. I’d love to hear about your experiences. There’s no comments section here unfortunately, but feel free to reach out on X.
