Sheriff Statistics for June 2017

July 3rd, 2017 by cbook


Welcome to the Sheriff Statistics for June 2017 !

Also i would like to thank everyone for taking part in the Sheriff Survey – You can see the results here :
Now to the actual data for June!
June 2017:

Autoland Tree:

Total Servo Sync Pushes: 254
Total Pushes: 1799
Total Number of commits 3711
Total number of commits without Servo 3445
Total Backouts: 167
Total of Multi-bug pushes 12
Total number of bugs changed 1702
Percentage of backout against bugs: 9.81198589894
Percentage of backouts: 9.28293496387
Percentage of backouts without Servo: 10.8090614887 (thats ~ +0,8 % higher rate compared to may)

Total Servo Sync Pushes: 0
Total Pushes: 1117
Total Number of commits 3611
Total number of commits without Servo 3611
Total Backouts: 130
Total of Multi-bug pushes 159
Total number of bugs changed 1591
Percentage of backout against bugs: 8.17096165933
Percentage of backouts: 11.6383169203
Percentage of backouts without Servo: 11.6383169203 (~ +0,7 % higher rate compared to may)

So Sheriffs managed and monitored on the Integration Trees in May 2017 ~ 2900 pushes and 297 backouts.

Let us know when you have any Question or Feedback about Sheriffing.

Cheers and have a great July!,

Sheriff Survey Results

June 23rd, 2017 by cbook
first a super big thanks for taking part in this years Sheriff Survey – this helps us a lot !
Here are the results.
1. Overall “satisfaction” – we have asked how People rate their interaction with us (from 1 (bad) to 10 (best)
So far from all results:
3,1 % = 5
3,1 % = 7
12,5 % = 8
43,8 % = 9
37,5 % = 10
2. What can we do better  as Sheriffs?
We got a lot of Feedback thats its not easy to find out who is on “sheriffduty”. We will take steps (like adding |sheriffduty tag to irc names etc) also we have with the target to have that name on treeherder.
Also we try to make sure to Set Needinfo requests on Backouts.
In any case, backouts are never meant to be personal and it’s part of our job to try our best to keep our trees open for developers. We also try to provide as much information as possible in the bug for why we
backed out a change.
3. Things we can improve in general (not just sheriffs) ?
A interesting Idea in the Feedback we got was about Automation. We will follow up from the Feedback and i already filed for the Idea of having a “Backout Button” in Treeherder in case no Sheriff is around etc – more bugs from ideas to improve general stuff and workflows will follow.
Again thanks for taking part in the Survey and if you have questions/feedback/concerns/ideas you can of course contact me / the team at anytime !
– Tomcat

Reminder :) Please take part in the Sheriff Survey!

June 19th, 2017 by cbook

just a reminder that we have our Sheriff Survey Running and please take part in it, it helps us a lot to improve our work!





Sheriff Survey 2017 started !

June 7th, 2017 by cbook

One thing i love at Mozilla is that there is no general “that’s the way we have always done it..” – we at Mozilla try to improve things – our Software but also our processes and so its also with Sheriffing – We try to optimize our workflow/processes constantly to make your live as developer easier when working with us but … we rely also on your feedback 🙂

So we created a Survey for YOU to give us Feedback about how we do and also where we can improve. This input/Feedback from you is very valuable for us!

You can find the Survey at

Thanks for taking part in the Survey!

– Tomcat

Sheriffing@Mozilla – Sheriffing and Backouts

April 3rd, 2017 by cbook


Keeping the code trees [1] green (meaning free of build or test failures,
regressions, and minimizing intermittent test failures) is the daily
goal of sheriffing.

In order to reach this goal, this means we sometimes have to back out (revert)
changes made by developers. While this is a part of our job, we don’t do
it easily or without reason.

Backouts happen mostly for:
-> Bustage (i.e. Firefox no longer
successfully builds)
-> Test failures caused by a specific change
-> Issues reported by the community, like startup crashes or severe
regressions (these backouts often lead to new nightly builds being
created as well)
-> Performance regressions or memory leaks
-> Issues that block merges like merge-conflicts (like for a mozilla-inbound to mozilla-central merge)

For our primary integration repositories (where our developers land most
their changes), our workflow depends on which repository the problem is


-> Close Mozilla-Inbound if needed (preventing
developers from landing any further changes until the problem is

-> Try to notify the responsible developer so that they
are  aware of the problem caused by their patch

-> If possible, we
accept follow-up patches to fix the problem. This allows us to fail
forward and avoid running extra jobs that require more CPU time and
therefore increase costs.

-> If we don’t get response from the developer within a short
timeframe like 5 minutes, we back out the change and comment in the
bug with a reason for the backout (for example, including a link to the
failure log) and a needinfo to the assigne, to make sure the bug don’t get lost.


-> Changesets that cause problems are backed out immediately –
no follow-ups as described above are possible (only the sheriffs can push manually to

In any case, backouts are never meant to be personal and it’s part of
our job to try our best to keep our trees open for developers. We also
try to provide as much information as possible in the bug for why we
backed out a change.

Of course, we also make mistakes and it could be that we backed out
changesets that were innocent (like in a case where its not 100% clear
what caused the problem), but we try our best.

If you feedback or ideas how we can make things better, let me know.

– Tomcat


[1] Trees: The tree contains the source code as well as the code required to build each project on supported platforms (Linux, Windows, macOS, etc) and tests for various areas. Sheriffs take care of Firefox Code Trees like mozilla-central, mozilla-inbound, autoland, mozilla-aurora, mozilla-beta and mozilla-esr45/52 – our primary tool is treeherder and can be found here

Sheriffing @ Mozilla – Sheriffing a Community Task!

February 27th, 2017 by cbook
i was recently asked if volunteers can help with Sheriffing!
And the answer is very simple: Of course you can and you are very welcome! 
As every part of Mozilla, volunteers are very important. Our team is mixed of Full-Time Employees and Volunteers.
What is needed to join as Community Sheriff:
I think basically there are 3 things you need to have to participate as Community Sheriff:
-> Communication Skills and Teamwork – Sheriffing means a lot of communication – communication with the other sheriff Teams, developers and teams like Taskcluster and Release Engineering. 
-> Background Knowledge how Bugzilla works (commenting in bugs, resolving bugs and setting flags etc)
-> Ability to see context & relationships between failures (like the relation of a set of failures to a checkin) to identify the changeset that causes the regression.
All our tools are public accessible and you don’t need any specific access rights to get started.
Our main tool is Treeherder ( and the first task a Community Sheriff could do is to watch Treeherder and star failures.
We have described this task here
That would help us a lot!
When you are curious how a day in Sheriffing looks then maybe can help 🙂
Please let us know when you are interested in becoming a Sheriff! You can find us on in the #sheriffs channel!

Sheriffing @ Mozilla – checkin-needed

January 26th, 2017 by cbook
Working as Sheriff @ Mozilla is much more than just monitoring our trees and doing things like backouts. In 2017 i wanted to start to blog more about what we do and here is:
Part 1 – Checkin-needed
A lot of checkins land everyday on the Mozilla repositories. Some are great new features and improvements and some are bugfixes of existing bugs etc.
While a lot of checkins are done by the developers themselves, also sheriffs are involved in this.
We not only monitor the Mozilla repositories (aka the tree) we also do checkins for people who don’t have the appropriate level of permission to check in changes (for example new community members).
In the past checkin-needed was used by developers to reduce load on our build systems with fewer pushes but with a more robust build system this isn’t relevant anymore. 
So checkin-needed is more and more important for developers without access-levels to do commits and, as mentioned, new community members who for example finished their first patch.
To request checkin-needed people use this keyword in Bugzilla or use the [checkin-needed-beta] or [checkin-needed-aurora] whiteboard entry for the patches in a bug.
For me personal is checkin-needed a very important task because you sometime check in a patch from someone who just started to contribute to Mozilla. So you are one of the people that are the first contacts to the new contributor and you help them getting the patch landed. Thats also a good opportunity to say “thanks for contributing to Mozilla” to the new community member, this is great motivation and recognition! 
How we work :
We have a wiki page with a bug query and some basic information for the Sheriff on Duty [1]. We use this query to get a overview what bugs need checkins.
For bugs with a patch attached that is not on mozreview we check the checkin-neeed request for:
    -> Has proper review before doing anything else
    -> Has a successful try run to avoid any bustage on checkin
and land the patch on mozilla-inbound.
For bugs with patches in mozreview we use the autoland tool to do the checkins. 
However we still check if the bug has review and check the try run.
This is the preferred way of doing checkin-neededs since autoland is an automated system.
How can you help 
  • ->  Use Mozreview – Autoland – it helps us to do more checkins in less time due to the automated tasks. Please make sure that there are no open issues in Mozreview when you request checkin-needed. In fact, you can land them yourself with autoland. In the future, checkin-needed will only be allowed on security bugs.
  • -> Make sure that you have a passing try run. It is a waste of the sheriffs time to come look at a checkin-needed and it has failures in the try run.
  • -> When you have multiple patches that need to land and the patches need to land in a specific order – please make a comment in the bug with the correct order.
  • -> When there are dependencies with other bugs – please state this in the bug.  
We try to do checkin-needed checks and checkins several times a day depending on sheriff workload etc so we cannot guarantee a turnaround time but trying to do our best.
When you have feedback/suggestions or idea how to do this task better let us know anytime!
Also as every part of the Mozilla Project we also depend on Community Members like you! So if you are interested to be become a Community Sheriff let me know!
– Tomcat

Please take part in the Sheriff Survey!

June 8th, 2016 by cbook


When we moved to the “inbound” model of tree management, the Tree Sheriffs became a crucial part of our engineering infrastructure. The primary responsibility of the Sheriffs is and will always be to aid developers to easily, quickly, and seamlessly land their code in the proper location(s) and ensure that code does not break our automated tests.

But of course there is always room for improvements and ideas how we can make things better. In order to get a picture from our Community (YOU!) how things went and how we can improve our day-to day-work we created a Survey!

You can find the Survey here:

Thanks for taking part in this survey!

Also you can find some of us also in London during the Mozilla All-hands if you want to talk to us directly!


– Tomcat

Sheriff Newsletter for January 2016

February 5th, 2016 by cbook
To give a little insight into our work and make our work more visible to our Community we decided to create a  monthly report of what’s going on in the Sheriffs Team.
If you have questions or feedback, just let us know!
In case you don’t know who the sheriffs are, or to check if there are current issues on the tree, see:
Topics of this month!
1. How-To article of the month
2. Get involved
3. Statistics for January
4. Orange Factor
5. Contact
1. How-To article of the month and notable things!
-> In the Sheriff Newsletter we mentioned the “Orange Factor” but what is this ?  It is simply the ratio of oranges (test failures) to test runs. The ideal value is, of course, zero.

Practically, this is virtually impossible for a code base of any substantial size,so it is a matter of policy as to what is an acceptable orange factor.

It is worth noting that the overall orange factor indicates nothing about the severity of the oranges. [4]

The main site where you can checkout the “Orange Factor” is at  and some interesting info’s are here
-> As you might be aware Firefox OS has moved into Tier 3 Support [5] – this means that there is no Sheriff Support anymore for the b2g-inbound tree.

Also with moving into tier-3 – b2g tests have also moved to tier 3 and this tests are by default “hidden” on treeherder. To view test results as example on treeherder for mozilla-central you need to click on the checkbox in the treeview “show/hide excluded jobs”.

2. Get involved!
Are you interested in helping out by becoming a Community Sheriff? Let us know!
3. Statistics
Intermittent Bugs filed in January  [1]: 667
and of those are closed: 107 [2]
For Tree Closing times and reasons see:
4. Orange Factor
Current Orangefactor [3]: 12.92
5.  How to contact us
There are a lot of ways to contact us. The fastest one is to contact
the sheriff on duty (the one with the |sheriffduty tag on their nick
🙂 or by emailing sheriffs @ mozilla dot org.

– Tomcat on the behalf of the Sheriffs

7 Years at Mozilla!

July 3rd, 2015 by cbook


since last month i’m now 7 years at Mozilla as full-time employee \o/

Of course I’m longer around because i started as Community Member in QA years before. And its a long way from my first steps at QA to my current role as Code Sheriff @ Mozilla.

I never actively planned to join the Mozilla Community it just happened 🙂 I worked back in 2001 at a German Email Provider as 2nd Level Support Engineer and as part of my Job (and also to Support Customers) we used different Email Programm’s to find out how to set-up the Programm and so.

Some Friends already involved into OpenSource (some linux fans) pointed me to this Mozilla Programm (at that time M1 or so) and i liked the Idea with this “Nightly”. Having everyday a brand new Program was something really cool and so started my way into the Community without even knowing that i’m now Part of the Community.

So over the years with Mozilla i finally filed my first bug and and was scared like hell (all this new fields in a non-native language) and not really knowing what i signed up when i clicked up this “submit” button in bugzillla 🙂  (was not even sure if i’m NOW supposed to fix the bug 🙂

And now i file dozens of Bugs every day while on Sheriffduty or doing other tasks 🙂

I learned a lot of stuff over the last years and still love being part of Mozilla  and its the best place to work for me! So on to the next years at Mozilla!

– Tomcat