Merge dates vs release dates

Christian Legnitto

The schedule on the rapid release process specifics document generally focuses on merge dates. There is some confusion as to what to expect on those dates, so hopefully this post will make it clear.

The main takeaway is that the merge date is not necessarily the date users on a particular update channel will see an update available.

Vastly simplifying, there are four general steps that need to happen before Mozilla ships browser bits to users:

  1. The source is ready/available in the proper place
  2. A build/update is created from the source and automatically placed on ftp.mozilla.org by the build system
  3. The build/update is qualified (either by automation or the QA team)
  4. The build/update is posted for users to easily get (on the update channel for updates and mozilla.com for full versions)

The different channels have differing quality expectations and thus the above steps are emphasized differently.

mozilla-central (Nightly)

  1. The source is ready/available in the proper place
    Nothing needs to be done for this step as mozilla-central is the reference/where the main source lives.
  2. A build/update is created from the source and automatically placed on ftp.mozilla.org by the build system
    This happens every night via automation because Nightly testers expect the latest fixes and the builds are unsigned (signing takes manual work).
  3. The build/update is qualified (either by automation or the QA team)
    There is no manual qualification on Nightly deliverables because there are no expectations of quality. We do bake a little stability in by making the automation from step #2 choose a changeset that passes automated tests.
  4. The build/update is posted for users to easily get (on the update channel for updates and mozilla.com for full versions)
    As above, there are no expectations of quality so this is automatically done at the same time as #2.

Synopsis: Because of the lower quality expectations, tech savvy audience, and fewer active users the updates for Nightly are available to users on that channel as soon as they are generated.

mozilla-aurora (Aurora)

mozilla-aurora is an entirely different beast:

  1. The source is ready/available in the proper place
    We have to merge source from mozilla-central to mozilla-aurora, which isn’t atomic and takes some time. The amount of time depends on the particular changes and how many conflicts we get. The schedule posted tells when the merge starts. The merge should only take a most a couple of hours but there is always the potential of it taking a day or so.
  2. A build/update is created from the source and automatically placed on ftp.mozilla.org by the build system
    This happens via automation the night after the merge is completed. We may manually start the process earlier if the merge is finished earlier.
  3. The build/update is qualified (either by automation or the QA team)
    For the first Aurora build after a merge we will take a small amount of time–likely a day but at most a week–to have QA manually qualify the update to make sure we didn’t screw up in the merge and strand our testers on a build that can’t update itself. Please note this isn’t tons of QA testing as we are relying on automated tests and the fact that Aurora users have explicitly self-selected into a potentially unstable channel.
  4. The build/update is posted for users to easily get (on the update channel for updates and mozilla.com for full versions)
    This doesn’t happen right when the build is generated in #2 because the qualification in #3 takes some small amount of time

Synopsis: Because the mozilla-aurora source merge is manual and the first update needs a bit of manual testing there is a lag between the merge date (reflected in the process specifics document) and the time Aurora users see an update available.

mozilla-beta (Firefox Beta)

mozilla-beta is similar to mozilla-aurora:

  1. The source is ready/available in the proper place
    We have to merge source from mozilla-aurora to mozilla-beta, which isn’t atomic and takes some time. The amount of time depends on the particular changes and how many conflicts we get. The schedule posted tells when the merge starts. The merge should only take a most a couple of hours but there is always the potential of it taking a day or so.
  2. A build/update is created from the source and automatically placed on ftp.mozilla.org by the build system
    This happens manually as Firefox Beta builds are signed, which requires human intervention.
  3. The build/update is qualified (either by automation or the QA team)
    For the first Beta build after a merge we will take some amount of time–likely less than a week–to have QA manually qualify the update. This testing is more rigorous than Aurora’s testing as the amount of people running a Firefox Beta is substantial. The testing should still be fairly quick as the bits have been tested on Aurora for 6 weeks at the mozilla-beta merge point…but we’re still treading a little more carefully here.
  4. The build/update is posted for users to easily get (on the update channel for updates and mozilla.com for full versions)
    This doesn’t happen right when the build is generated in #2 because the qualification in #3 takes some small amount of time

Synopsis: Because the mozilla-beta source merge is manual and the first update needs a bit of manual testing there is a lag between the merge date (reflected in the process specifics document) and the time Firefox Beta users see an update available.

mozilla-release (Firefox)

mozilla-release is different than the others because we have publicly committed to release dates (June 21st specifically for Firefox 5).  To the world “released” means the bits are available to install, not that the release process has started and the bits will come out within a day or so. This means the parts of the process that take additional time will overlap with the Firefox beta period and the final date in the process specifics document is the date the update is released not the merge date (which is at some currently undefined earlier date).

  1. The source is ready/available in the proper place
    We have to merge source from mozilla-beta to mozilla-release. This is an implementation detail. Firefox Beta manual builds point by default to the beta channel and we need to change that for release. We are currently not able to do channel repacks, so we do one additional build after beta to change the default channel to release. In an ideal world this rebuild wouldn’t happen but we aren’t there yet.
  2. A build/update is created from the source and automatically placed on ftp.mozilla.org by the build system
    This happens manually as Firefox builds are signed, which requires human intervention.
  3. The build/update is qualified (either by automation or the QA team)
    For Firefox releases we will take some amount of time–likely less than a week–to have QA manually qualify the update. This testing is rigorous.
  4. The build/update is posted for users to easily get (on the update channel for updates and mozilla.com for full versions)
    This doesn’t happen right when the build is generated in #2 because the qualification in #3 takes some small amount of time

Synopsis: The date for mozilla-release in the process specifics document is a real release date rather than a source merge date. It is set in stone. There is some manual mechanical work that needs to be done before the release date. That work will overlap with the Firefox Beta period.