At a meeting a couple of weeks ago, I volunteered to look at the Talos data for the last release cycle to see how stable the tests were.\u00a0 I wasn’t going to do extensive statistical analysis on the numbers, since the Talos servers already do that when deciding whether a change is significant or not.\u00a0 What I initially came up with was:<\/p>\n
(I realize that the specifics of the changesets involved may invalidate the generalizations made above.\u00a0 But as a good first cut, e.g. for somebody who’s just trying to see if there are significant Talos regressions, this seems like a good start.)<\/p>\n
I started out trying to catalog these numbers manually and quickly decided that was for the birds.\u00a0 I found myself mixing up numbers for different tests, different trees, and different platforms (to say nothing of PGO versus non-PGO builds on inbound!), due mostly to the large number of emails to look at and the similarity between the subjects of the emails.\u00a0 A better approach was called for.<\/p>\n
So I wrote talos-summarize<\/a> for parsing dev.tree-management archives for some date range and generating useful visualizations from them.\u00a0 Wanting some insight into the questions above, I chose a table whose rows represent ranges of changesets and whose columns contain data for how a particular platform’s numbers change.\u00a0 This makes it fairly easy to address point 1 above; point 2 is somewhat addressed by eyeballing the table and looking at the distribution of +\/-s, but it should also be addressed by computing cumulative changes from the beginning of the chosen range.\u00a0 That last point hasn’t been implemented yet.<\/p>\n Showing pictures would be more useful; I generated visualizations for all the Talos tests that ran on inbound<\/a> during the start of the Firefox 15 release cycle.\u00a0 The correspondence between the filenames and the tests themselves should be pretty obvious.<\/p>\n I initially started looking at the Ts, MED Dirty Profile table<\/a>.\u00a0 This table made me happy, because you can see the consistent jump across all platforms as a result of bug 769960<\/a> landing and the corresponding decrease when the startup regression from that bug landed, bug 778855<\/a>.\u00a0 That test also looks like it has fairly stable numbers; the numbers don’t jump around too much.\u00a0 The table also suggests that we significantly regressed startup time on multiple platforms with this pushlog<\/a>; this regression didn’t get addressed during the initial cycle of 15, even though it was about as severe as the regression from bug 769960 (and touched our more critical Windows platforms, too).<\/p>\n For the most part–and this is just eyeballing–the numbers from all tests don’t jump around or provide contradictory information.\u00a0 The one exception would be Tp5 No Network Row Major MozAfterPaint<\/a> (what a mouthful!), where the row in the middle looks quite odd.\u00a0 Unfortunately, stable numbers across all the tests also look like constantly increasing numbers across all the tests, which is not great.<\/p>\n