{"id":557,"date":"2026-04-21T17:40:11","date_gmt":"2026-04-22T00:40:11","guid":{"rendered":"https:\/\/blog.mozilla.org\/performance\/?p=557"},"modified":"2026-04-22T09:22:16","modified_gmt":"2026-04-22T16:22:16","slug":"telemetry-alerting-how-it-works","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/performance\/2026\/04\/21\/telemetry-alerting-how-it-works\/","title":{"rendered":"Telemetry Alerting: How It Works"},"content":{"rendered":"

We recently released the telemetry alerting beta, and announced it in the blog post here<\/a>! This blog post will dive into the details of how it works across Treeherder, and Mozdetect. At a high level, MozDetect handles the change point detection for telemetry probes, and Treeherder handles storing the detections, and producing the emails\/bugs for these.<\/p>\n

MozDetect<\/h2>\n

All of the existing, and any future change detection point techniques used for telemetry alerting are built in MozDetect<\/a>. Having these live outside of Treeherder gives a low-barrier to entry for adding new features, and testing existing ones without having to set up everything needed for alerting in Treeherder. It\u2019s built as a python module that is run through uv<\/a>. This makes it very easy for anyone to run the code because of uv\u2019s excellent python version, and dependency management. How to work with the code in this repository is outlined here<\/a>, along with how to add your own techniques to it<\/a> (note the access to mozdata through gcloud is required for this).<\/p>\n

Detectors are split into two parts: (i) a detector that performs a comparison between two groups, and (ii) a detector that performs detection on a time series (using the detector from (i)). Our default detection technique, called \u00a0cdf_squared<\/code>\u00a0 lives here<\/a>. The\u00a0 timeseries_detector_name<\/code>\u00a0 is the name that will be used to access the detector from the telemetry probe side through the\u00a0 change_detection_technique<\/code>\u00a0 field. The only method that absolutely needs to be implemented by these is the detect_changes<\/a> method and it must return a list of Detection<\/a> objects. These detection objects contain all the necessary information for producing an alert. There is also an\u00a0 optional_detection_info<\/code>\u00a0 field that can contain additional things like attachments<\/a> that would be added to Bugzilla bugs, and additional_data<\/a> that can hold JSON data for storage in the DB. The cumulative distribution function<\/a> (CDF) squared technique uses these to store the CDF before and after the detection along with a graph of these as an attachment for the Bugzilla bug<\/a>.<\/p>\n

\"\"

Example of a CDF graph that is provided in bugs.<\/p><\/div>\n

CDF Squared Detection Technique<\/h2>\n

The CDF squared technique detects changes in time-series histogram data by comparing CDFs between consecutive windows. It takes two CDFs, each representing the distribution of measurements over a time window, and computes the sum of squared differences between the two CDFs at each bin. The sign of the summed linear difference is then used to assign a direction to the squared difference score so that the output encodes whether the distribution moved to higher values (right shift) or lower values (left shift).<\/p>\n

For time-series detection, this base comparison is applied in a rolling fashion across the full history of data. Each day’s 7-day smoothed CDF is compared against the next one, producing a continuous signal of squared CDF differences over time. A Butterworth low-pass filter<\/a> is then applied to that signal to remove high-frequency noise while preserving genuine trend changes. Finally, scipy\u2019s find_peaks<\/a> function is used to locate statistically significant peaks and valleys in the filtered signal using a dynamic alert threshold based on the historical data. Information is extracted from those areas and then used to build the detection information needed for the alert generation process.<\/p>\n

 <\/p>\n

Alerting<\/h2>\n

Our alerting tooling lives in the Treeherder codebase<\/a>. It\u2019s run through our PerfSheriff Bot (called Sherlock) and runs once per day. When a detection is produced from MozDetect, a telemetry alert is added to the database and then the TelemetryAlertManager<\/a> is called to handle it. The manager\u2019s tasks are split into 6 ordered phases<\/a>:<\/p>\n

    \n
  1. Update alerts with changes from Bugzilla.<\/b> This step ensures that any changes that happen in the bugs filed are mirrored into our database. Currently, we only track resolution changes here.<\/li>\n
  2. Comment on existing bugs<\/b>. This step is for updating existing bugs with information from new alerts. This step is not currently being used. In the future, this could be used to inform probe owners that a probe which doesn\u2019t produce bugs has produced an alert in the same time range.<\/li>\n
  3. File new bugs for alerts.<\/b> This step handles filing bugs for any new alerts on probes set up for producing bugs.<\/li>\n
  4. Modify existing bugs with new alerts. <\/b>This step handles any modifications needed to existing bugs based on the new bugs that were created. Currently, the “See Also” field is modified for existing bugs to include the new bugs.<\/li>\n
  5. Produce emails for new alerts.<\/b> This step handles producing emails for any alerts set up to produce emails.<\/li>\n
  6. Housekeeping.<\/b> This step handles redoing any failures that happen above in either the current run or past runs. Currently, it\u2019s being used to retry bug modifications and sending emails when we encounter a failure there. This excludes retrying bug filling since we delete the alert in that case and retry it the next time the alert is generated.<\/li>\n<\/ol>\n

    After the housekeeping step, the manager is done for the day and runs again on the next day to handle any updates and new alerts. Contrary to how alerting works for performance tests in CI, this process is fully automated and requires no human input at any point.<\/p>\n

    Setting up telemetry probes for alerting happens on the mozilla-central side in their probe schema using the new\u00a0 monitor<\/code>\u00a0 field in the\u00a0 metadata<\/code>\u00a0 section (example for email alerts<\/a>, example for bug alerts<\/a>). The telemetry alerting documentation<\/a> has information about how to do this. We then use an index.json<\/a> file from the telemetry dictionary to gather all the probes that should be alerting. The information there is supplemented by more granular information<\/a> later in the pipeline to gather things like the time unit used for the probe to be able to better format the Bugzilla bug table.<\/p>\n

    Once a telemetry probe is set up for alerting and is found by our system, the owners (those listed in the email notification fields) will begin either receiving emails or have bugs produced for them. These can also be viewed by everyone on this dashboard<\/a>.<\/p>\n

    \"\"

    Example of an alert being viewed in the dashboard<\/a>.<\/p><\/div>\n

     <\/p>\n

     <\/p>\n

    Acknowledgements<\/h2>\n

    Getting the project to this point involved work from people across multiple teams here at Mozilla. Special thanks to Eduardo Filho<\/strong> for his support on the telemetry probe side, to Bas Schouten<\/strong> for his guidance and work on the CDF Squared detection technique, and to Andrej Glavic<\/strong> and Beatrice Acasandrei<\/strong> for their help in reviewing the Treeherder-related changes.<\/p>\n

    If you hit any issues with the telemetry alerting system, or have any suggestions feel free to file a bug in the Testing :: Performance component<\/a> or reach out to us in either #perf-help on Slack or in #perftest on Matrix<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

    We recently released the telemetry alerting beta, and announced it in the blog post here! This blog post will dive into the details of how it works across Treeherder, and … Read more<\/a><\/p>\n","protected":false},"author":1804,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[457027],"tags":[457028,457016],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts\/557"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/users\/1804"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/comments?post=557"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/posts\/557\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/media?parent=557"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/categories?post=557"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/performance\/wp-json\/wp\/v2\/tags?post=557"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}