rstrong's blog
in search of ponies
Why data is good and how it helped me deal with Firefox Frankenbuilds
April 14th, 2011 by rstrong
The updater has always had a problem on Windows (likely other platforms as well) where an update doesn’t update all of the files. The latest occurrence of this is on the Firefox 3.6 branch in Bug 639542 – Crash [@ js_DeepBail(JSContext*) ] at address 0×204 due to frankenbuild. Anurag Phadke was able to get data out of http://crash-stats.mozilla.com/ that allowed us to evaluate how bad the problem was and from that analysis I filed bug 635161 to change the updater’s behavior and bug 635834 to get more data so it is possible to evaluate where we are now and so it is possible to evaluate the impact, if any, of bug 635161 after it has made it to an official release.
Below is a simple analysis from the data… long story short, with Firefox 3.5 approximately 8% of crashes contained mismatched dll’s, Firefox 3.6 approximately 0.25% of crashes contained mismatched dll’s, and Firefox 4.0 (includes nightly) approximately 0.04% of crashes contained mismatched dll’s. If the Firefox 4 data is restricted to just the last month the percentage drops down to approximately 0.02% of crashes contained mismatched dll’s. This is by no means declaring success… we definitely can do better.
Total crashes processed : 31761889
Firefox 3.5 Mismatched File Version Report
20101204 through 20110411
Summary
=============================
Lines processed : 1711982
Crashes processed : 1721575
Lines skipped : 361
Total mismatches : 137788
Percent mismatches : 8.00%
Skipped Line Breakdown
===============================
Invalid firefox.exe name : 309
Missing version info : 52
Mismatched Versions
==============================================================================+
| Total | DLLVer < FFVer | DLLVer > FFVer |
+------------------+-----------------+----------------+
Total mismatches : 137788 (100.00%) | 135850 (98.59%) | 1938 (1.41%) |
browserdirprovider.dll : 13507 (9.80%) | 11575 (8.40%) | 1932 (1.40%) |
brwsrcmp.dll : 6087 (4.42%) | 4894 (3.55%) | 1193 (0.87%) |
xpcom.dll : 101843 (73.91%) | 101810 (73.89%) | 33 (0.02%) |
xul.dll : 135854 (98.60%) | 135823 (98.57%) | 31 (0.02%) |
+------------------+-----------------+----------------+
Firefox 3.6 Mismatched File Version Report
20101204 through 20110411
Summary
==============================
Lines processed : 20277061
Crashes processed : 20278718
Lines skipped : 3797
Total mismatches : 49995
Percent mismatches : 0.25%
Skipped Line Breakdown
================================
Invalid line format : 34
Invalid firefox.exe name : 2907
Missing version info : 857
Mismatched Versions
==============================================================================+
| Total | DLLVer < FFVer | DLLVer > FFVer |
+------------------+-----------------+----------------+
Total mismatches : 49995 (100.00%) | 1039 (2.08%) | 48956 (97.92%) |
browserdirprovider.dll : 48685 (97.38%) | 370 (0.74%) | 48315 (96.64%) |
brwsrcmp.dll : 48465 (96.94%) | 351 (0.70%) | 48114 (96.24%) |
xpcom.dll : 16297 (32.60%) | 426 (0.85%) | 15871 (31.75%) |
xul.dll : 7903 (15.81%) | 877 (1.75%) | 7026 (14.05%) |
+------------------+-----------------+----------------+
Firefox 4.0 Mismatched File Version Report
20101111 through 20110411
Summary
=============================
Lines processed : 9750994
Crashes processed : 9761596
Lines skipped : 954
Total mismatches : 3679
Percent mismatches : 0.04%
Skipped Line Breakdown
===============================
Invalid line format : 46
Invalid firefox.exe name : 246
Missing version info : 664
Mismatched Versions
==============================================================================+
| Total | DLLVer < FFVer | DLLVer > FFVer |
+------------------+-----------------+----------------+
Total mismatches : 3679 (100.00%) | 95 (2.58%) | 3584 (97.42%) |
browsercomps.dll : 3501 (95.16%) | 82 (2.23%) | 3419 (92.93%) |
mozalloc.dll : 2110 (57.35%) | 77 (2.09%) | 2033 (55.26%) |
xpcom.dll : 2067 (56.18%) | 75 (2.04%) | 1992 (54.15%) |
xul.dll : 2531 (68.80%) | 77 (2.09%) | 2454 (66.70%) |
+------------------+-----------------+----------------+
Now if only some data source could answer how the same thing can happen to an extension… I had a user who was definitely running a mixture of files from Adblock Plus 1.3.3 and Adblock Plus 1.3.5 – on Firefox 4 where Adblock Plus is supposed to be only one XPI file.
Just brought this up to Mossop (he sits next to me) and both him and myself are scratching our heads. Would be good to file an Add-ons Manager bug for this since it definitely shouldn’t happen.
A lot of the current FF4 users probably have manually downloaded and used the installer, I think a more fair picture of FF4 will come once we have an announced major update out.
Still, good work in reducing those problems!
Excellent point that I hadn’t thought about with everything else going on. Even before the release it was showing a reduction from approximately 160 in every 2000 to 1 in every 2000 of crashes so that is definitely a major improvement.
I filed that bug: https://bugzilla.mozilla.org/show_bug.cgi?id=650216
Not sure whether this is enough information to reproduce the issue but that’s all I’ve got.