Graph Server Re-Write

Over the past few months the Graph Server team and I have been hard at work re-writing the back end for the Graph Server and it’s finally come to fruition.

For those that don’t know, the Graph Server is used to display performance test data of Firefox builds reported by Talos.

Graph Server screenshot

Our work initially started as performance improvements and some new features, but the more we worked with the old architecture, it became quite apparent it would not scale (performance and feature-wise).

The old database schema duplicated test data in multiple tables and stored similar, but different data in the same tables. Tables had ballooned to millions (and billions) of rows that were queried for basic information such as all unique test names, resulting in queries that ran forever. And the queries that did finish were looped over in JavaScript to pull out test information, resulting in the browser locking up because it was looping over hundreds of thousands of rows.

If it’s not clear already, one of main issues was with the database schema; it needed to be normalized.

Here’s the old, non-normalized schema:

Old Graph Server schema

And here’s the new, normalized schema after the team was locked in a room for an afternoon:

New Graph Server schema

Much cleaner, no duplicated data, easy to understand the various machines, branches and tests that are used for displaying test data. No need look at entire tables to find basic information such as test names.

With this new schema in place, it also required a re-write of our server-side scripts we use to fetch test information for the front end graphing component. Since Mozilla is as open as possible, instead of just changing what was needed, I decided to implement a JSON API that would allow anyone to easily retrieve test data.

Lastly, our Talos <--> Graph Server communication needed to be re-written. Lars rewrote the collector script that accepts values from Talos and Alice rewrote the pieces of Talos that send data to the Graph Server.

After all that work, we now have a working stage server (Firefox 3.1 or higher required due to native JSON requirement) with our new code. We have a bit more testing and some performance benchmarking to do before it goes live, but we’re happy that all the pieces are working.

Want to know more? We have a wiki page with more information at https://wiki.mozilla.org/Perfomatic#Rearchitecture.

7 responses

  1. RyanVM wrote on :

    Yay, my browser doesn’t freeze when I close the tab now! Nice job!

  2. morgamic wrote on :

    Great work, everyone! You should post a link to the mac mini cluster. 🙂

  3. Tristan wrote on :

    A lot of improvement in terms of speed! Congrats!

  4. Paddy wrote on :

    I have a question, What modelling software is that?

  5. sikiÅŸ izle wrote on :

    RyanVM: Yay, my browser doesn’t freeze when I close the tab now! Nice job!

    I agree with you. I closed it too. And work now perfect..

    In article says; After all that work, we now have a working stage server (Firefox 3.1 or higher required due to native JSON requirement) with our new code. We have a bit more testing and some performance benchmarking to do before it goes live, sikiş izle but we’re happy that all the pieces are working..

    Please publish the firefox 3.1 or higher. I think now beta version available..

  6. G wrote on :

    yah, same question… those are spiffy ER diagrams… what’d you use to generate them?

  7. rdoherty wrote on :

    @Paddy & G:

    I used SQLEditor (http://www.malcolmhardie.com/sqleditor/) to create the diagrams.