I want more users on the Nightly channel

I have been working recently on a new Platform Engineering initiative called Uptime, the goal of which is to reduce Firefox’s crash rate on both desktop and mobile. As a result I’ve been spending a lot of time looking at crash reports, particular on the Nightly channel. This in turn has increased my appreciation of how important Nightly channel users are.

A crash report from a Nightly user is much more useful than a crash report from a non-Nightly user, for two reasons.

  • If a developer lands a change that triggers crashes for Nightly users, they will get fast feedback via crash reports, often within a day or two.  This maximizes the likelihood of a fix, because the particular change will be fresh in the developer’s mind. Also, backing out changes is usually easy at this point. In contrast, finding out about a crash weeks or months later is less useful.
  • Because a new Nightly build is done every night, if a new crash signature appears, we have a fairly small regression window. This makes it easier to identify which change caused the new crashes.

Also, Nightly builds contain some extra diagnostics and checks that can also be helpful with identifying a range of problems. (See MOZ_DIAGNOSTIC_ASSERT for one example.)

If we could significantly increase the size of our Nightly user population, that would definitely help reduce crash rates. We would get data about a wider range of crashes. We would also get stronger signals for specific crash-causing defects. This is important because the number of crash reports received for each Nightly build is relatively low, and it’s often the case that a cluster of crash reports that come from two or more different users will receive more attention than a cluster that comes from a single user.

(You might be wondering how we distinguish those two cases. Each crash report doesn’t contain enough information to individually identify the user — unless the user entered their email address into the crash reporting form — but crash reports do contain enough information that you can usually tell if two different crash reports have come from two different users. For example, the installation time alone is usually enough, because it’s measured to the nearest second.)

All this is doubly true on Android, where the number of Nightly users is much smaller than on Windows, Mac and Linux.

Using the Nightly channel is not the best choice for everyone. There are some disadvantages.

  • Nightly is less stable than later channels, but not drastically so. The crash rate is typically 1.5–2.5 times higher than Beta or Release, though occasionally it spikes higher for a short period. So a Nightly user should be comfortable with the prospect of less stability.
  • Nightly gets updated every 24 hours, which some people would find annoying.

There are also advantages.

  • Nightly users get to experience new features and fixes immediately.
  • Nightly users get the satisfaction that they are helping produce a better Firefox. The frustration of any crash is offset by the knowledge that the information in the corresponding crash report is disproportionately valuable. Indeed, there’s a non-trivial likelihood that a single crash report from a Nightly user will receive individual attention from an engineer.

If you, or somebody you know, thinks that those advantages outweigh the disadvantages, please consider switching. Thank you.

24 Responses to I want more users on the Nightly channel

  1. Forgive me if this sounds rude, but is Firefox for Android still being actively developed as though Android was a tier 1 platform?

  2. done! I used to use nightly exclusively, but stopped at one point i think due to annoyances with addons’s not updating. Now I use far fewer addons, so there’s nothing holding me back!

  3. I’ve been Nightly as my daily browser at home and at work for a long time now. I love it because it’s the browser where the all the new features appear first.
    I would greatly appreciate a site where new features are listed so as to be able to try them as soon as they are enabled though!
    It very seldom crashes in my experience and I don’t mind at all updating it daily!

  4. What’s the difference between Developer Edition and Nightly build? The prior one is also updated almost everyday.

    • Nicholas Nethercote

      There are four stages in the Firefox development cycle, each about 6 weeks long. There is six weeks of active development, then six weeks of stabilization on Dev Edition, then six weeks more stabilization on Beta, then six weeks on general Release. Nightly builds are made every night from the current in-development version, so it contains all the changes that developers have made that day. In contrast, the Dev Edition doesn’t see those changes until the 6 week cycle ends.

      Also, although Dev Edition is updated every night, typically the number of changes made each night is small, and those changes are only things like bug fixes.

  5. I tried switching. What I found is that installing and using Nightly is unnecessarily cumbersome because biggest showstopper: (by default) You cannot run it at the some time as Firefox stable.

    So I did not really use Nightly because of just this. Eventually I found that you can get this working if you manually create another profile. Nightly is now my fallback browser.

    I filed this bug for this and hope you make using Nightly as easy it is to run the Dev Edition: https://bugzilla.mozilla.org/show_bug.cgi?id=1265322

    • To give some feedback. Next to this, my problems are:

      * Broken Addons. Most potential Nightly users are powerusers, that is, have quite a few addons. However, many of these are broken on Nightly, often even in the Dev Edition. In my case, it’s the ‘brief’ rss reader that I heavily use and that’s not working on anything else but stable.

      * For Ubuntu there is no ppa, that installs Nightly *next* to stable. There is a ppa for nightly, but this overwrites the stable version then (https://launchpad.net/~ubuntu-mozilla-daily/+archive/ubuntu/ppa). For the dev edition there is Ubuntu make: http://askubuntu.com/a/584704 I do not see an equivalent for Nightly?!

      Anyway, I installed it at least on Android, where I do not have many addons except ublock origin and adblock, but Firefox 49 let’s me block those with the “Tracking protection” (which is also available in not-private mode now) so that’s really nice now and I do not need these anymore.

  6. Ah, fine you’ve convinced me! I had previously avoided nightly on android as I found the daily updates tedious. I also had to stop using nightly on desktop due to frequent mission-critical addons breaking (Zotero), but this won’t be a problem on mobile.

  7. I’ve been using Nightly on Android for at two years as my main mobile browser and it’s totally OK. I very rarely switch to Firefox stable or to Chrome if something doesn’t work (sometimes it helps, sometimes it doesn’t). It used to crash very frequently, but now even that is rare. I am happy to report bugs and send crash reports, and I’m happy that somebody reads them.

    And you know what would make even more happy? If I’d get a note in the email that says something like this: “We analyzed your crash report. Here’s the Bugzilla link for the bug that we opened as a result.”

    Sometimes I open a bug myself, if I see a recurring reproducible crash, but usually I just send the report, and get on with my life and web browsing, with the hope that it’s helpful to somebody. Actually knowing that *my* crash report helped fix a *particular* issue would be much more motivating, even if I don’t really understand the underlying technical issue.

    • +1

      This is even true for the Desktop Firefox. If I am honest, I found myself even not sending crash reports because it seems like sending them to a big black hole.

      • Nicholas Nethercote

        Crash reports are very useful, and people look at them all the time.

        As for the email idea… hmm. For people who put their email address into the crash report, that’s one of the most privacy-sensitive pieces of information in the crash report. So we are *really* careful with it. While you might like to be contacted in this way, I can imagine that other people might not. So in general I’d rather err on the side of caution.

        • It’s a tradeoff.

          If I enter my mail address, where it also clearly says that I might be contacted – to which I agree -, then I don’t see why this should not happen. I definitely wish it would! If I give you my mail address, than you have it anyway. Why not contact me? Please. Especially for nightly (testing) users who consider themselves affiliated to mozilla.

          I’d consider it to be rewarding if I knew that my report was helpful and ended in a (fixed) bug report. Today, it feels like sending a letter to the editor, that never gets printed 😉

  8. Happy nightly user here.
    The frequent updates are not a problem, it’s just more free stuff: http://www.stickycomics.com/computer-update/

  9. I used to run nightly on Android all the time, until I got a new phone and Sync was broken, so I couldn’t sync my bookmarks and passwords anymore (https://bugzilla.mozilla.org/show_bug.cgi?id=1138943 and https://bugzilla.mozilla.org/show_bug.cgi?id=1204559). If these would be fixed, I would use nightly again. For the rest, I really enjoyed nightly on Android and I also use it on all my desktops.

  10. I thought, that testing of beta is more useful, but you would like to have more Nightly testers, so that I switched to Nightly channel.
    Now you have one more Nightly tester 🙂

    • Nicholas Nethercote

      Thank you for switching. Beta testers are good, but Nightly testers are better 🙂

  11. Ciprian Khlud

    For me it is important to know that FF should not crash, and I tried to use Nightly but kind of often FFN freezes. Maybe is extension related (extensions like LastPass do go offline), but is very annbe oying nonetheless.

    I would say that creating a minimal web service on Mozilla side (maybe in partnership with some trusted entity) to make a subset of extensions working and being kind of supported in nightly would be a really great

    I know that maybe it feels a lot of harsh criticism, and I’m sorry if this seem to be the tone. As a developer myself, sometimes I don’t know how the sea of bugs go fixed.

    A last note, FF itself in its Nightly form it is definitely very stable, but I will try to spend from time to time 1 hour of typical usage to get from time to time crash reports to you.

  12. Two things. I’m seeing crashes on Firefox for Android that aren’t getting caught. This is not only on my devices running Nightly but devices running RTM as well. I’m partly at fault because I haven’t submitted any catlogs but this is also stuff we should expect QA to pick up way before it ever gets to RTM.

    Secondly, Nightly for Ubuntu needs a Mozilla controlled PPA. I’ve not seen a build since 2016/04/01. That hampers testing in a huge way.

  13. I have been using FDE as my daily browser for several months now–it has only been since then, that it has been stable on my systems (or with my addons?). I have two questions.

    (1) I am willing to give Nightly a shot, given what you’ve just said—but would you say that Nightly surfing is more important than surfing with FDE?

    Also, I don’t submit bug reports. I tried in the past, but they were always duplicates, so I simply wasted everyone’s time (or it was an addon, or something like that).

    So (2) would it be worth your while to have users surf on Nightly, WITHOUT submitting bug reports?

    • Actually, re-reading your post, it looks like the answer to both questions is “Yes.” I should try re-reading a bit more often. My initial read is more like a scan. 🙁

    • Nicholas Nethercote

      Yes, Nightly is better for helping reduce the crash rate than the Developer Edition, because of the immediate feedback and small regression windows.

      Using Nightly without submitting bug reports is definitely still useful, because you can still submit crash reports. (If you don’t submit crash reports, that is much less useful.)

      Filing a bug report is most useful if you have clear steps to reproduce. That’s something that is not present in a crash report, and it makes a huge difference.