Update on Add-on Performance Testing

Justin Scott

17

Just over a week ago we announced our add-on performance initiative and have received lots of feedback from add-on users and developers. Thanks to our awesome community, many developers have updated their add-ons to have faster start-up time, and others have dug into the results and filed bugs to help us improve our testing framework. Our automated tests don’t work perfectly with every add-on, so if you aren’t able to reproduce our results, please file a bug and let us know.

One of the most popular components of our performance initiative is the Slow Performing Add-ons page. We’ve made some changes to the page to more accurately describe the performance results shown and hide add-ons that have minimal impact on performance from being included in the list. Now, only add-ons with an impact of 25% or more are displayed instead of an ordered list of all add-ons tested. Additionally, we’ve corrected some wording that referred to these add-ons as the “slowest”. We apologize for this oversight.

We initially planned to begin displaying warnings for add-ons that add more than 25% to start-up time this week, but will delay that until we can verify and fix any major issues discovered by the community. However, all of the add-ons currently displayed on the page have been verified as causing a significant impact on Firefox start-up through manual testing and real-world data, in addition to the automated testing.

As we said in the announcement, this is only the beginning of our work to improve and educate about add-on performance, and we’ll continue improving our tools and documentation to help developers.

17 responses

  1. Kohei Yoshino wrote on :

    Japanese translation of this article is here:
    https://dev.mozilla.jp/2011/04/update-on-add-on-performance-testing/

  2. Wladimir Palant wrote on ::

    “fix any major issues discovered by the community” – like the use of nonsensical percentages that are 2-3 times higher than what the user should expect to see?

    1. Justin Scott (fligtar) wrote on ::

      Which add-ons have percentages that are 2-3 times higher than what users will see?

      1. Wladimir Palant wrote on ::

        All of them. These percentages are based on average Firefox startup times of 500 ms. What you get in the real world however is cold startup (that’s what users mainly complain about) and non-clean profile state, so 1 or 1.5 seconds for Firefox startup while add-on initialization times typically don’t change.

        It’s not like this is the first time I explain that. See bug 648742, nobody replied there so far.

        1. Jorge Villalobos wrote on ::

          I guess the blog ate my previous comment…

          This is the cold vs warm startup argument, which has also been discussed among ourselves. While I generally agree that cold startup overhead is a more realistic metric, I don’t think that warm startup overhead measurements are meaningless or nonsensical.

          The difference would be that the overhead percentages would be lower, as well as our expectations. If we currently set the bar at 25% overhead for warm startup, then it would probably be around 10% for cold startup, and I doubt the list of slow add-ons would be any different. Yes, the percentages would be less dramatic and some users and the press would be less shocked by them, which I know is not a trivial concern.

          However, we don’t have a system in place to properly measure cold startup performance, so we use what we have. It’s definitely worthwhile looking into it and move in that direction as quickly as possible, but dismissing the results outright as nonsensical seems counterproductive to me.

          1. Nils Maier wrote on ::

            The result percentages and rankings are indeed nonsensical, so dismissing them is not counterproductive, but actually the right and only thing to do here.

          2. Wladimir Palant wrote on ::

            Jorge, I don’t see you find a way to get realistic results any time soon (other than throwing Talos out of the window and using ping data instead). It’s not only cold startup vs. warm startup, it is also profile state – a clean profile allows for much faster startup times than one with a properly filled places database (there are probably also other contributing factors). Which is why you shouldn’t use that metric in the first place – absolute numbers seem way more suitable in helping users make a decision. You admit it yourself – the current numbers are only suitable for shocking users and press which will eventually result in killing off every complicated add-on out there.

            And I don’t really care where you put the arbitrary boundary at which you call add-ons “slow”. This is a pointless discussion right now given that you don’t have enough usable data to base a decision on. Once the most serious bugs in Talos measurements are fixed this can be looked into again.

      2. Wladimir Palant wrote on ::

        PS: You own data (http://people.mozilla.com/~fligtar/performance/num-addons-med.png) suggests that real-world Firefox startup times are even significantly beyond 2 seconds. Does the impact of add-ons also increase by factor 4-5 on consumer machines? I very much doubt that.

  3. Harsh86 wrote on :

    25% is kinda high. It should be 20% or even 15% IMO.

    1. Wladimir Palant wrote on ::

      Instead of 25% one could also say: 130 milliseconds (0.13 seconds). Still too high?

      1. ancestor wrote on :

        Absolute numbers vary depending on hardware so I think percentages are the way to go. However, I agree they must be based on more realistic scenarios.

        1. ancestor wrote on :

          I haven’t read your latest blog post when I wrote that. If you are right that the add-on slowdown is not proportionate to Firefox’s overall startup time, it is indeed an argument against percentages. However, this doesn’t change the fact that absolute numbers also can be misleading. So the question is, which sucks less?

          1. Wladimir Palant wrote on ::

            You cannot make it right for everybody. However, so far my impression is that absolute numbers allow better decisions. I don’t think that they vary too much on current consumer hardware (worth investigating but I can only test on hardware that I have in my household). And somebody using a netbook or a 10 years old PC will know to expect higher numbers than for the “reference platform”. Right now the effect is that *everybody* expects much too high slowdown numbers which is IMHO unacceptable.

  4. Michael Lefevre wrote on :

    The previous comments have already more than covered the point I was going to make, which is that 25% sounds horrible when your start up time might be 5-20 seconds already. If, as Wladimir seems pretty clear on, those percentages are nothing like what the average user is going to see in a cold start with a well-used profile, then it would make sense to give some real numbers instead of percentages…

  5. Mark Smith wrote on ::

    For moderately sophisticated users, reporting the times in milliseconds rather than as percentages would be much more useful.

    But the biggest problem I see is that the page at https://addons.mozilla.org/performance strongly implies that add-ons that increase Firefox startup time also slow down Firefox all the time as you use it. Is there any data to back that up? If not, the wording on the page should be changed, disclaimers added, etc.

    1. Johan Sundström wrote on ::

      There’s nothing about this data that relates to run-time speed.

      On the contrary, add-ons like AdBlock Plus (with, say, the EasyList filter), which were present in the first version of the table, make your daily browsing significantly faster than a naked Firefox installation.

      While a step in the right direction, the presentation of this data isn’t the best.

      Also, any add-on that stores a disproportionate amount of data (in total) in prefs.js will slow down start-up times very significantly. Example: if you run a few Greasemonkey scripts that all make heavy use of GM_setValue (which stores string keys in prefs.js), to store, say, some few hundred kb together, startup time gets completely unbearable.

  6. Ed Edwards wrote on :

    QA QA QA QA QA & QA
    Any quantitative analysis should be WELL received and welcomed. Perfection is hard to achieve…