Untangling the terminology

In the old days, before DVCS, “commit” only had only one real purpose. It was how you published your work to the rest of the world (or your project’s world at least). With DVCS, you are likely committing quite often, but still only occasionally publishing.

In the early days of hg at Mozilla, you would “push your changes”. But with the advent of social coding sites such as bitbucket & github, you often push your changes there as well. Or you push to our try servers.

Informally, Mozillian’s use the term “landing” to describe committing a change to one of the official repositories (also known as “repository of record” or RoR). So while most of the online documentation refers to “commit”, I’ll use the less ambiguous term “landing” here.

The Evolvolution of the Landing Cycle

It happened slowly, so we may not have noticed it, but the distinctions between “commit” and “land” have changed over time. Can we benefit from embracing these changes and extending them further? I believe so, even though that might require us to view our toolsets differently.

So What?

Besides demonstrating my skill level at graphviz, what can be learned from these diagrams? I think something very special:

There are two separate work flows here, and they interact only in well defined ways.

or

The work flow of a development is distinct from the workflow of landing changes into the official repositories.

I assert that we can make the following conclusion from that observation:

This really shouldn’t be news worthy – it is actually the way things have always been. Before DVCS, every developer did their own source code management on their computer. Maybe it was multiple checkouts, tarballs, patch sets, but chances are it had nothing to do with the tool used for the RoR. Developers even managed to exchange information with each other, again via tarballs, emailed patches, directory permissions and a host of other methods outside of the RoR tooling. (Personally, my first production use of hg was to follow the pattern Mozilla documented for using hg with CVS. That was wonderfully freeing, as it gave me the safety net of local commits and local branches. As a developer, I wanted and could make good use of far more flexibility than any well designed release system would let me have.)

Introduction

To start things off, here are the simplified workflows that describe how things currently work with the Mercurial repositories on hg.m.o and following commit policy. The major differences between the committer’s workflow and the developer’s workflow are:

• the source of the changes (could be patches from developers without commit access, in addition to development work done directly by the committer).
• the ultimate destination repository is the repository of record hosted on hg.mozilla.org.

The following table should match today’s workflow for landing someone else’s (or their own) approved patch:

Committer Action	Steps Involved	Pseudo Commands
First time checkout [2]
	get personal copy
	get copy on PC	hg clone hg.m.o PC
	tweak for 1 step commit to hg	add hg.m.o as pushable path
Work Cycle
	incorporate r+ work [3]	hg patch OR pull from github or bitbucket (may not match r+ attachment) [4]
	resolve conflicts
	local work	hg commit
Commit Process
		hg push hg.m.o

Changes for Bitbucket

As bitbucket uses mercurial as it’s primary DVCS, you might expect there to be not too many changes. That’s correct – there is just some additional steps on bitbucket so other users can see your work.

The changes from hg.m.o are italicized.

Committer Action	Steps Involved	Pseudo Commands
First time checkout [2]
	get personal copy	bitbucket fork
	get copy on PC	hg clone bitbucket PC
	tweak for 1 step commit to hg	add hg.m.o as pushable path
Work Cycle
	incorporate r+ work [3]	hg patch OR pull from github or bitbucket (may not match r+ attachment) [4]
	resolve conflicts
	local work	hg commit
Commit Process
		hg push hg.m.o

Changes for Github

For github, there are a few additional steps that will be supported by scripts. As above, the differences from hg.m.o are italicized.

Committer Action	Steps Involved	Pseudo Commands
First time checkout [2]
	get personal copy	github fork
	get copy on PC	git clone github PC
	tweak for 1 step commit to hg	run script to setup hg repo & git post commit hooks & magic_refs [5]
Work Cycle
	incorporate r+ work [3]	apply patch manually OR pull from github or bitbucket (but may not match r+ attachment) [4]
	resolve conflicts
	local work	git commit
Commit Process
		git push magic_ref [5]

Introduction

To start things off, here are the simplified workflows that describe how things currently work with the Mercurial repositories on hg.m.o and following commit policy.

This should match today’s workflow for devlopment (not commit):

Developer Action	Steps Involved	Pseudo Commands
First time checkout
	get copy on PC	hg clone hg.m.o PC
	tweak for 1 step push to try	add try as pushable path
Work Cycle
	local work [1]	hg commit
	push to try [1]	hg push magic_try_path
Review Process
	create patch	attach patch to bug
Commit Process
	nag committer

Changes for Bitbucket

As bitbucket uses mercurial as it’s primary DVCS, you might expect there to be not too many changes. That’s correct – there is just some additional steps on bitbucket so other users can see your work.

The changes from hg.m.o are italicized.

Developer Action	Steps Involved	Pseudo Commands
First time checkout
	get personal copy	fork on bitbucket
	get copy on PC	hg clone bitbucket PC
	tweak for 1 step push to try	add try as pushable path
Work Cycle
	local work [1]	hg commit
	push to try [1]	hg push magic_try_path
Review Process
	create patch	attach patch to bug OR hg push bitbucket and bitbucket pull request
Commit Process
	nag committer

Changes for Github

For github, there are a few additional steps that will be supported by scripts. As above, the differences from hg.m.o are italicized.

Developer Action	Steps Involved	Pseudo Commands
First time checkout
	get personal copy	github fork
	get copy on PC	git clone github PC
	tweak for 1 step push to try	run script to setup hg repo & git post commit hooks & refs [2]
Work Cycle
	local work [1]	git commit
	push to try [1]	git push magic_try_remote
Review Process
	create patch	attach patch to bug OR git push github and github pull request
Commit Process
	nag committer

Summary

A long time ago (December of 2011), I sent out a brief survey on DVCS usage to Mozilla folks (and asked them to spread wider). While there were only 42 responses, there were some interesting patterns.

I believe you can see the raw summary at via this link. What follows are the informal inferences I drew from the results (remember the disclaimer).

The Issue Is…

The issue is some level of official support of social coding sites:

• 86% mentioned "pull requests" as a better review process than patch submittal.
• A number of volunteered comments expressed some level of confusion, including which github repo to use, how to start a new project, what workflows to use to interact with hg.m.o, etc.

A Plan

These survey results, plus some technical investigation into what was possible with given tools, led to forming the (very agressive) 2012 Q1 goal captured in bug 713782. A looser summary of that follows.

Where we are in January 2012

The purpose of this post is to present a very high level picture of the current Firefox build & release process as a set of requirements. Some of these services are provided or supported by groups outside of releng (particularly it & webdev). This diagram will be useful in understanding the impact of changes.

System Scope

Here is how I’m defining the boundaries of the current system:

• Applies to anything & everything involved in producing artifacts that are directly installed on an end user’s device.
  • In: Firefox, Thunderbird, Seamonkey
  • Not in: services such as browser id, sync, the addons.m.o., apps, and other code that executes on Mozilla’s servers.
  • Not in: extensions themselves (they are installed into FF/TB, not directly onto the end user’s device). Yes, that is a hair I want to split. [1]
• But only if the entire process happens "inside the Mozilla firewall". (For example, the download mirrors are not in scope.)

[1]	There may be some extensions, such as the Thunderbird Lightening, that cross this line. For now, I’ll ignore those.

The Developer Services

N.B. these are only repository related services that a devloper commonly accesses.

With this setup, there is official support for:

• pulling from Mercurial repositories hosted at mozilla.org (hg.m.o)
• commit access from developer’s own clone of the Mercurial repository (with committer policy enforcement)
• "try server" access from developer’s own clone of the Mercurial repository
• continuous integration based on commits to hg.m.o
• tracking of results of commit compilation & testing, via tbpl.m.o.
• web viewing of repositories
• web cross reference of many repositories on mxr.m.o

The Picture

The Ideal Future

The Ideal: In the future, each contributor will be as free to choose their DVCS of choice as they currently are to choose a text editor.

Since that’s a pretty general statement, let me limit it a bit with some caveats that I expect will apply.

• This may only be true at a technical level, as various teams might have social conventions or alternate tooling that will limit DVCS choices.
• Initial planning will be for products [1] that are released via the Release Engineering team. [2]
• The current set of supported DVCS would be hg and git. By supported, I think folks roughly mean:
  • an officially supported way get the latest revision of any official branch/repository, without any extra work on the part of the developer.
  • an officially supported way to interact with existing Mozilla hosted tools, such as the try server, tbpl, etc.
  • an officially supported way to commit from the DVCS, that supports the current commit workflow (and policy).
• There will be support of social coding sites. I think support means providing an officially supported mechanism to keep "current" instances of each official branch in the social coding sites. The goal would be to allow developers to utilize the enhanced services offered by those sites to view, clone, and share work prior to doing an official commit.

Next Steps

1. This post will help validate that the above vision has a wider consensus. Please pass this link along and add your comments below.
2. Keep an eye on the main page for more material.

---

[1]	(1, 2) I’m using "products" to distinquish binaries Mozilla provides to the end users to install on their computers from "services" Mozilla hosts. Product examples include Firefox and Thunderbird. Service examples include Sync and BrowserID.

[2]

The rationale for starting with existing products is that is where the majority of our developers work. Additionally, the existing continuous build & test infrastructure will place some constraints on DVCS workflows. Once those constraints are documented, other projects can make informed decisions on their usage.

Posts in the Series

Overview & context:

• Where we want to end up
• Where we are today
• What’s the first step?
• Other benefits to the first step

Implementation Details:

• Changes to developer workflow
• Changes to committer workflow

Glossary

The project goal (above) has a few terms that need defining, so they mean the same thing to all of us. And the discussion will introduce some terms as well. As these come up, they will be added here.

Mozilla Products

Software delivered to the end user for installation directly on their computer.

Releng Infrastructure

The processes & machines that are used to produce Mozilla Products, from the source code repository through continuous integration and testing to the product binaries ready for the end user.

Related & Reference Material

• Master tracking is occurring in bug 713782
• John O’Duinn’s post "is it git or is it github"

What folks have already done:

• doublec’s mozilla-central on github https://github.com/doublec/mozilla-central
• bholley & jlebar’s tools to easy committing to hg from git https://github.com/jlebar/moz-git-tools

Summary

So things are different at Mozilla, but isn’t that always the case when changing jobs? Well, yes, but Mozilla appears to be more different:

• We’re guided by our Mission Statement.
• We produce open source products.
• We’re owned by a not for profit foundation.
• Much that most companies would consider “trade secrets” or “proprietary information” about process or plans, we discuss in public.

But, how different does this make the inner workings of Mozilla? As a release engineer, how would my day-to-day work change? How much of what-I-thought-I-knew need to be thrown out? Would I be putting myself into an “everything you know is wrong” situation? [2]

I’m sure I’ll be dealing with this cultural shift for at least my first year. I’m looking forward to this, as it will be a wonderful opportunity to examine a number of assumptions I’ve mad for years, that may no longer be useful. (And, even if one assumption is discarded, I don’t know if it will be replaced by one that contradicts the original, or subsumes the original with a more general understanding of the process.)

For example, one such question I’m currently pondering is: what is the role of the VCS (version control system) in an environment like Mozilla? Does it differ from private enterprise? If so, how? This question is one I’m keenly interested in, both philosophically, and practically. As is often the case, writing about it clarified some things for me. I hope folks who comment will help polish it further.

What Services are provided by a VCS

One might (and I suspect many readers will) think this is a silly question. A VCS exists to support developers. Period. Historically, that’s a very valid point. If one is just looking at software development, we can put the development team in the center of everything:

Most development operations now have a number of automated systems which interact with the VCS, including:

• defect tracking
• continuous integration
• metrics (code churn, bug density analysis, etc)

Toughest of all to spot are the non-automated policies that rely on certain behavior being stable over a long period of time. These implicit contracts impact operations such as:

• legal audit trails [3]
• disaster recovery plans
• the big boss’s custom metrics script that assumes things don’t change.

I won’t attempt to draw it, but now imagine that every data store and automated process in those diagrams need to be replicated or clustered for scaling and HA. All of a sudden, our nice neat little VCS system has gotten out of hand. Any change has to meet the needs of the entire user community, not just the developers.

---

[1]	Here I’m using “product” to mean something that is produced my Mozilla and installed on end user devices. (Firefox, Thunderbird, Seamonkey are all examples.) Mozilla also provides services, which may have different needs.

[2]	This is what programmers were told when they started to program the first Macintosh, as it required a change from procedural flow to event driven flow. Of course, that was a huge lie, as many programming fields already used event driven programming. (Even RPG II, but that’s another story.)

[3]	Mozilla has these – the commiter’s agreement combined with commit level authorization