{"id":29,"date":"2017-07-18T11:28:55","date_gmt":"2017-07-18T11:28:55","guid":{"rendered":"https:\/\/blog.mozilla.org\/data\/?p=29"},"modified":"2019-10-17T20:20:44","modified_gmt":"2019-10-17T20:20:44","slug":"firefox-data-platform-tools-update-q2-2017","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/data\/2017\/07\/18\/firefox-data-platform-tools-update-q2-2017\/","title":{"rendered":"Firefox data platform & tools update, Q2 2017"},"content":{"rendered":"

The data platform and tools<\/em> teams are working on our core Telemetry system, the data pipeline, providing core datasets and maintaining some central data viewing tools.<\/p>\n

To make new work more visible, we provide quarterly updates.<\/p>\n

What\u2019s new in the last few months?<\/strong><\/p>\n

\"Beta

Beta \u201cmain\u201d ping submission delay analysis by :chutten.<\/p><\/div>\n

A lot of work in the last months was on reducing latency, supporting experimentation and providing a more reliable experience of the data platform.<\/p>\n

On the data collection<\/strong> side, we have significantly improved reporting latency from Firefox 55, with preliminary results from Beta showing we receive 95% of the \u201cmain\u201d ping within 8 hours<\/a> (compared to previously over 90 hours). Curious for more detail? #1<\/a> and #2<\/a> should have you covered.<\/p>\n

We also added a \u201cnew-profile\u201d ping<\/a>, which gives a clear and timely signal for new clients.<\/p>\n

There is a new API to record active experiments in Firefox<\/a>. This allows annotating experiments or interesting populations in a standard way.<\/p>\n

The record_in_processes<\/em> field is now required for all histograms. This removes ambiguity about which process they are recorded in.<\/p>\n

The data documentation<\/strong> moved to a new home: docs.telemetry.mozilla.org<\/a>. Are there gaps in the documentation you want to see filled? Let us know by filing a bug<\/a>.<\/p>\n

For datasets<\/strong>, we added telemetry_new_profile_parquet<\/em>, which makes the data from the \u201cnew-profile\u201d<\/em> ping available.<\/p>\n

Additionally, the main_summary<\/em> dataset now includes all scalars and uses a whitelist for histograms, making it easy to add them. Important fields like active_ticks<\/em> and Quantum release criteria were also added and backfilled.<\/p>\n

For custom analysis<\/strong> on ATMO<\/a>, cluster lifetimes can now be extended self-serve in the UI. The stability of scheduled job stability also saw major improvements.<\/p>\n

There were first steps towards supporting Zeppelin notebooks better; they can now be rendered as Markdown<\/a> in Python.<\/p>\n

The data tools<\/strong> work is focused on making our data available in a more accessible way. Here, our main tool Redash<\/a> saw multiple improvements.<\/span><\/p>\n

Large queries should no longer show the slow script dialog<\/a> and scheduled queries can now have an expiration date. Finally, a new Athena data source was introduced, which contains a subset of our Telemetry-based derived datasets. This brings huge performance and stability improvements over Presto.<\/p>\n

What is up\u00a0next?<\/h3>\n

For the next few months, interesting projects in the pipeline include:<\/p>\n