Tomcat's Blog

Hi,

Welcome to the Sheriff Statistics for June 2017 !

Hi,
just a reminder that we have our Sheriff Survey Running and please take part in it, it helps us a lot to improve our work!

Link: https://docs.google.com/a/mozilla.com/forms/d/e/1FAIpQLSfGBZ50zkG9W-Wnk1ACBfFvj1iu8e46I5gs9t-G3ZWDpcy4-A/viewform

thanks!

Tomcat

Hi,

Keeping the code trees [1] green (meaning free of build or test failures,
regressions, and minimizing intermittent test failures) is the daily
goal of sheriffing.

In order to reach this goal, this means we sometimes have to back out (revert)
changes made by developers. While this is a part of our job, we don’t do
it easily or without reason.

Backouts happen mostly for:
-> Bustage (i.e. Firefox no longer
successfully builds)
-> Test failures caused by a specific change
-> Issues reported by the community, like startup crashes or severe
regressions (these backouts often lead to new nightly builds being
created as well)
-> Performance regressions or memory leaks
-> Issues that block merges like merge-conflicts (like for a mozilla-inbound to mozilla-central merge)

For our primary integration repositories (where our developers land most
their changes), our workflow depends on which repository the problem is
on.

Mozilla-Inbound

-> Close Mozilla-Inbound if needed (preventing
developers from landing any further changes until the problem is
resolved)

-> Try to notify the responsible developer so that they
are  aware of the problem caused by their patch

-> If possible, we
accept follow-up patches to fix the problem. This allows us to fail
forward and avoid running extra jobs that require more CPU time and
therefore increase costs.

-> If we don’t get response from the developer within a short
timeframe like 5 minutes, we back out the change and comment in the
bug with a reason for the backout (for example, including a link to the
failure log) and a needinfo to the assigne, to make sure the bug don’t get lost.

Autoland

-> Changesets that cause problems are backed out immediately –
no follow-ups as described above are possible (only the sheriffs can push manually to
autoland)

In any case, backouts are never meant to be personal and it’s part of
our job to try our best to keep our trees open for developers. We also
try to provide as much information as possible in the bug for why we
backed out a change.

Of course, we also make mistakes and it could be that we backed out
changesets that were innocent (like in a case where its not 100% clear
what caused the problem), but we try our best.

If you feedback or ideas how we can make things better, let me know.

Cheers,
– Tomcat

[1] Trees: The tree contains the source code as well as the code required to build each project on supported platforms (Linux, Windows, macOS, etc) and tests for various areas. Sheriffs take care of Firefox Code Trees like mozilla-central, mozilla-inbound, autoland, mozilla-aurora, mozilla-beta and mozilla-esr45/52 – our primary tool is treeherder and can be found here

Hi,

When we moved to the “inbound” model of tree management, the Tree Sheriffs became a crucial part of our engineering infrastructure. The primary responsibility of the Sheriffs is and will always be to aid developers to easily, quickly, and seamlessly land their code in the proper location(s) and ensure that code does not break our automated tests.

But of course there is always room for improvements and ideas how we can make things better. In order to get a picture from our Community (YOU!) how things went and how we can improve our day-to day-work we created a Survey!

You can find the Survey here:

Hi,

since last month i’m now 7 years at Mozilla as full-time employee \o/

Of course I’m longer around because i started as Community Member in QA years before. And its a long way from my first steps at QA to my current role as Code Sheriff @ Mozilla.

I never actively planned to join the Mozilla Community it just happened 🙂 I worked back in 2001 at a German Email Provider as 2nd Level Support Engineer and as part of my Job (and also to Support Customers) we used different Email Programm’s to find out how to set-up the Programm and so.

Some Friends already involved into OpenSource (some linux fans) pointed me to this Mozilla Programm (at that time M1 or so) and i liked the Idea with this “Nightly”. Having everyday a brand new Program was something really cool and so started my way into the Community without even knowing that i’m now Part of the Community.

So over the years with Mozilla i finally filed my first bug and and was scared like hell (all this new fields in a non-native language) and not really knowing what i signed up when i clicked up this “submit” button in bugzillla 🙂  (was not even sure if i’m NOW supposed to fix the bug 🙂

And now i file dozens of Bugs every day while on Sheriffduty or doing other tasks 🙂

I learned a lot of stuff over the last years and still love being part of Mozilla  and its the best place to work for me! So on to the next years at Mozilla!

– Tomcat

Hi,

we closed our Sheriff Survey on Monday and i wanted to share some highlights from the Results. Thanks for taking part in the Survey!

1.Communication with the Sheriffs

We got very good and positive Feedback about the Interaction/Communication with the Sheriffs. We know that backouts are never a good/positive thing and we sheriffs assume always the best intentions – nobody _wants_ to cause bustage, but it happens.

We also noticed a lot of comments of checkin-needed requestors and the hope we have at some time the autolander system (that lands patches automatically). There is work being done on this like as example in https://bugzilla.mozilla.org/show_bug.cgi?id=1128039

2. Trychooser and other Feedback

We got comments about trychooser and how this could be improved. That Feedback is very valuable and we will pass that Feedback over to the Releng Folks. For all Feedback and Suggestions we are looking at the survey what we can improve and realize. As example one result is now https://bugzilla.mozilla.org/show_bug.cgi?id=1145836 🙂

3. Getting Involved!

We got several Community Member with interest in helping out with Sheriffing! Thats really great and we will follow-up here soon. Also its not too late to get involved. Just drop me or the sheriff lists (sheriffs@mozilla.org) a note!

4. You can reach us at anytime!

While the Survey is closed now you can still contact us anytime for feedback, questions and when you want to be involved! Just drop us a note at sheriffs@mozilla.org or ping the Sheriff on duty (normally the one with the |sheriffduty tag in #developers on irc.mozilla.org).

Thanks!

– Tomcat

Hi,

thanks for all the Reply’s we got for the Sheriff Survey! If you haven’t already took part in it, its still online and you can still take part in the survey!

While we close the Survey in a few days and i will provide a comprehensive overview of course, i was feeling i could already do some quick overview what we got so far.

One big take away is how important checkin-needed requests is and how many people depend on this. We are very sorry if there are delays with picking up checkin-needed requests but since its a human task it depend how much is ongoing with the trees etc.

But there is work being done on Autoland like on https://wiki.mozilla.org/Auto-tools/Projects/Autoland 🙂

Also to follow up on 2 concrete things (you might know or maybe not).

Question: How do i know why the tree is closed (when we have a tree closure) on Treeherder

Answer:  Just hover over the repo name in Treeherder (as example mozilla-inbound) or click on the info button right next to the repo name

Question: When i land something on like mozilla-inbound its a mess to manually copy and past the hg changeset url to bug

Answer: We have a tool called mcmerge its right next to every push in the drown-down arrow action menu and unlike the name says its not just to mark merges. During the survey we found out that the name is misleading so we trying to find a new name – https://bugzilla.mozilla.org/show_bug.cgi?id=1145836

Thanks,

– Tomcat

Hi,