Unbundling MediaStreamTrackProcessor and VideoTrackGenerator

Earlier this year we promised more blog posts about new and exciting WebRTC functionality in standards. So today we’re talking about MediaStreamTrackProcessor and VideoTrackGenerator! These two APIs let web developers convert a real time video MediaStreamTrack to and from a stream of VideoFrames respectively. This simplifies real time video processing in JavaScript.

These APIs are video-only and worker-only which avoids jank when scrolling by never blocking the browser’s real time media pipeline on a busy main thread — They’re available in Safari 18 Tech Preview and will be in Firefox by the beginning of next year.

Chrome on the other hand, shipped proprietary versions of these APIs back in 2021. Those are instead exposed on the main-thread, and also handle audio. This is obviously not great for web interoperability. We’ll look at both versions, and propose solutions for web developers wondering how to manage this discrepancy.

Both versions promise convenience and performance. But are they equally necessary? And what about audio? Let’s explore.

(more…)

How WebRTC speaker selection works

Today’s topic is speaker selection. If you’ve never given it much thought, or consider it straightforward, this post is for you. Let’s start by testing your general WebRTC knowledge:

Pop Quiz: If a website wants to play out of different speakers on your system, what permission must it have?

1. 1. Speaker-selection permission
   2. Microphone permission

If you answered 2 then chances are you know your WebRTC stuff well, but you’re probably on a Chromium browser.

Pause for a moment. Why would websites need microphone permission to control speaker output? Why should users expose themselves to possibly being recorded just to redirect songs to their portable speakers? They shouldn’t. It’s a permission escalation and an entirely unnecessary invasion of privacy. Nevertheless, in some browsers this remains the only answer today.

Thankfully, Firefox provides the more straightforward and privacy-preserving answer of 1.

In this post, we’ll look at how speaker selection works in different browsers, how permissions can be abused for fingerprinting purposes, and how to add speaker selection with a fallback so it works in all browsers.
(more…)

Debugging with about:webrtc in Firefox, Getting Data Out

Helpful Firefox about: pages for debugging WebRTC

Debugging WebRTC calls can be difficult. When you work on WebRTC any call you are in can unexpectedly become a troubleshooting session and printing messages to the JS console may not cut it. Firefox ships with several tools to help. We’ll briefly touch on each of these tools at a glance and learn when you might want to reach for them.

Using about:webrtc and about:logging

about:webrtc has lots of information about WebRTC sessions. Of all the tools covered here, this one is the most tailored for debugging active call issues. You can look at the call signaling, connection establishment, network traffic, and more. In a future article we will look at effective ways to get data out of about:webrtc and into your notes, bug reports, and more. There are more generalized tools available as well, so let’s take a peak at those next.
(more…)

setCodecPreferences is now in all browsers!

WebRTC enables video calls directly in browsers. Effectively managing the codecs that encode and decode media streams is a crucial component of delivering high-quality audio and video. The setCodecPreferences method allows developers to specify which codecs their applications prefer to receive.

With Firefox Nightly 128, setCodecPreferences is now available in all browsers! Let’s explore how it functions, its implications for web developers, and its role in harmonizing browser behaviors.

Firefox supporting this WebRTC 1.0 feature is a big deal. It helps webpages work the same across all browsers, eliminating browser-conditional code or having to resort to munging WebRTC’s signaling messages written in SDP, by hand.

But what made it seem blog-worthy was this question: do people know how it works? Looking at MDN and recent spec changes suggested not. Contrary to popular belief, it doesn’t control what you send. At least not directly.
(more…)

End-to-end-encrypt WebRTC in all browsers!

Hi all, we haven’t blogged since the WebRTC API was standardized back in January of 2021! But now in 2024 folks are asking us what’s up with all the new APIs? — We’re coming off an exciting second period of innovation and experimentation in WebRTC, so we’re brushing off the old blogging muscles to cover what’s new, in what’s promising to be a new series of posts.

First, don’t worry. WebRTC 1.0 has been tremendously successful and has not fundamentally changed. Instead, its success has led to developers wanting to use it for a variety of things. This demand has driven continued collaboration in the W3C  to standardize new functionality on top of the 1.0 baseline. Each new API exposes exciting new functionality, and each one deserves its own blog post, so let’s start right away with one of them:

Today’s topic is End-to-End-Encryption (E2EE) in WebRTC.

First, we should note that WebRTC is already peer-to-peer encrypted by DTLS, so there’s no need to use additional APIs except to protect data against any middle-boxes your service might employ as WebRTC end-points. That said…

Did you know all major browsers have an API to encrypt WebRTC calls end-to-end? They do! The caveat is the API shape differs slightly between browsers right now. But don’t despair! We’re here to cover that gap. If you’ve been with us for a while you know we’ve been here before, and this is how the sausage is made.
(more…)

Removing Old Versions of DTLS

As you probably have read already old versions of TLS are going to be removed in March 2020. Obviously we don’t want to leave Firefox WebRTC users behind and vulnerable. Thus we are planing on disabling support for DTLS 1.0 also in March 2020.

According to our measurements the absolute majority (98.12%) of WebRTC services use DTLS 1.2 already today.

The remaining 1.88% need to start upgrading to DTLS 1.2 today.

To help with the transition and testing we added new user preferences to Firefox 71 (currently available as Firefox Beta):

• media.peerconnection.dtls.version.min = 770
• media.peerconnection.dtls.version.max = 771

To test with DTLS 1.2 only simply open about:config and set media.peerconnection.dtls.version.min to 771. From now on your Firefox will no longer offer DTLS 1.0, as if it would be a release from March 2020.

Camera & microphone require https in Firefox 68.

As of Firefox 68—releasing next week—camera and microphone will require an https connection to work. Access from insecure http will cease to work, matching how Chrome works. Camera and microhone are powerful features, and Firefox will now only expose them in secure contexts, a security and privacy improvement.

Largely an issue for developers to catch up on, this may nonetheless affect end users who follow old http links to servers that either still don’t redirect their users to https automatically, or don’t support it at all. If you’re experiencing this problem, click your URL bar, and if the URL starts with http, try changing it to https, and it should work again, provided the server supports https.

The rest of this blog post is for web developers, since there are some details that may be of interest to them. Firefox is deprecating camera and microphone access in insecure contexts in two steps: (more…)

Perfect negotiation in WebRTC

New preface: What if you could add and remove media to and from a live WebRTC connection, without having to worry about state, glare (signaling collisions), role (what side you’re on), or what condition the connection is in? You’d simply call pc.addTrack(track, stream) regardless of time and place, and your track would just show up on the opposite side, without risk of terminal connection failure. A pipe dream? Too much to ask? Actually, now that Chrome has finally fixed its negotiationneeded event, this almost works! But you’re not using it, because “almost” doesn’t cut it. It only works 95% of the time, and the stakes from glare are too high in Chrome (no way to roll back from it, just pc.close()). But this original promise of the API is still within reach. We just need browsers other than Firefox to implement rollback, and fix some glaring (no pun intended) races in a few methods in the specification.

Perhaps few things in the WebRTC API cause as many pilot errors as the asymmetric exchange of SDP and ICE. Its stateful signalingState and timing sensitive ICE trickling can be a source of races if programmed incorrectly. As if that’s not challenging enough, keeping the two sides in sync and the two directions apart throws a lot of people for a loop (more puns).

What if I told you we can tame this complexity? That the manual way most people go about negotiating in WebRTC today is inferior, and possibly racy?

I’m talking about negotiationneeded and "rollback". Two mechanisms that can be used in concert to abstract negotiation away entirely. Yet, no-one is using them, because they haven’t worked, and don’t work yet in Chrome, respectively. We’ll introduce both. (more…)

isRemote in getStats() will disappear in Firefox 66

Update: We’ve pushed this back to Firefox 66, giving you a little more time!

Please go to isRemote in getStats() will disappear in Firefox 66.