Sheriffing@Mozilla – Sheriffing and Backouts

April 3rd, 2017 by cbook

Hi,

Keeping the code trees [1] green (meaning free of build or test failures,
regressions, and minimizing intermittent test failures) is the daily
goal of sheriffing.

In order to reach this goal, this means we sometimes have to back out (revert)
changes made by developers. While this is a part of our job, we don’t do
it easily or without reason.

Backouts happen mostly for:
-> Bustage (i.e. Firefox no longer
successfully builds)
-> Test failures caused by a specific change
-> Issues reported by the community, like startup crashes or severe
regressions (these backouts often lead to new nightly builds being
created as well)
-> Performance regressions or memory leaks
-> Issues that block merges like merge-conflicts (like for a mozilla-inbound to mozilla-central merge)

For our primary integration repositories (where our developers land most
their changes), our workflow depends on which repository the problem is
on.

Mozilla-Inbound

-> Close Mozilla-Inbound if needed (preventing
developers from landing any further changes until the problem is
resolved)

-> Try to notify the responsible developer so that they
are  aware of the problem caused by their patch

-> If possible, we
accept follow-up patches to fix the problem. This allows us to fail
forward and avoid running extra jobs that require more CPU time and
therefore increase costs.

-> If we don’t get response from the developer within a short
timeframe like 5 minutes, we back out the change and comment in the
bug with a reason for the backout (for example, including a link to the
failure log) and a needinfo to the assigne, to make sure the bug don’t get lost.

Autoland

-> Changesets that cause problems are backed out immediately –
no follow-ups as described above are possible (only the sheriffs can push manually to
autoland)

In any case, backouts are never meant to be personal and it’s part of
our job to try our best to keep our trees open for developers. We also
try to provide as much information as possible in the bug for why we
backed out a change.

Of course, we also make mistakes and it could be that we backed out
changesets that were innocent (like in a case where its not 100% clear
what caused the problem), but we try our best.

If you feedback or ideas how we can make things better, let me know.

Cheers,
– Tomcat

 

[1] Trees: The tree contains the source code as well as the code required to build each project on supported platforms (Linux, Windows, macOS, etc) and tests for various areas. Sheriffs take care of Firefox Code Trees like mozilla-central, mozilla-inbound, autoland, mozilla-aurora, mozilla-beta and mozilla-esr45/52 – our primary tool is treeherder and can be found here

736 Responses to “Sheriffing@Mozilla – Sheriffing and Backouts”

  1. upsc preparation strategy Says:

    upsc preparation strategy

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  2. Ismael Vidot Says:

    Ismael Vidot

    I found a great…

  3. Lecia Seidenbecker Says:

    Lecia Seidenbecker

    I found a great…

  4. Invisalign Calgary Says:

    Invisalign Calgary

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  5. Lorie Kochanski Says:

    Lorie Kochanski

    I found a great…

  6. Free Private Proxies For Sale Says:

    Free Private Proxies For Sale

    I found a great…

  7. Terina Zielonka Says:

    Terina Zielonka

    I found a great…

  8. Marquita Arechiga Says:

    Marquita Arechiga

    I found a great…

  9. Fabian Dinius Says:

    Fabian Dinius

    I found a great…

  10. Preparation of proposals Says:

    Preparation of proposals

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  11. Bestproxies Says:

    Bestproxies

    I found a great…

  12. Proxies For Seo Says:

    Proxies For Seo

    I found a great…

  13. Your Private Proxy Says:

    Your Private Proxy

    I found a great…

  14. Blazing Proxy Says:

    Blazing Proxy

    I found a great…

  15. Rita Lauffer Says:

    Rita Lauffer

    I found a great…

  16. выключатель ва 55 41 Says:

    выключатель ва 55 41

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  17. Madaline Wengert Says:

    Madaline Wengert

    I found a great…

  18. Anibal Contrell Says:

    Anibal Contrell

    I found a great…

  19. логистическая компания Says:

    логистическая компания

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  20. Caleb Henjes Says:

    Caleb Henjes

    I found a great…

  21. Usa Private Proxy Says:

    Usa Private Proxy

    I found a great…

  22. startup coach Says:

    startup coach

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  23. relevant resource site Says:

    relevant resource site

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  24. Bud Unterzuber Says:

    Bud Unterzuber

    I found a great…

  25. https://vimeo.com/tag:underground+hip+hop+producers Says:

    https://vimeo.com/tag:underground+hip+hop+producers

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  26. moneylender company Says:

    moneylender company

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  27. scrap car process Says:

    scrap car process

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  28. Sslprivateproxy Says:

    Sslprivateproxy

    I found a great…

  29. Gene Huffaker Says:

    Gene Huffaker

    I found a great…

  30. guideon Says:

    I have read so many posts regarding the blogger lovers however this article is in fact a good paragraph, keep it up

  31. Andy Offley Says:

    Andy Offley

    I found a great…

  32. personal cleansing wipes Manufacturers Says:

    personal cleansing wipes Manufacturers

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  33. f type jaguar цена Says:

    f type jaguar цена

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  34. vat registration documents Says:

    vat registration documents

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  35. PR Application Says:

    PR Application

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

  36. мебельный магазин Says:

    мебельный магазин

    Tomcat's Blog » Blog Archive » Sheriffing@Mozilla – Sheriffing and Backouts

Leave a Reply