After receiving numerous reports from add-on developers about slight dips in their usage statistics, we discovered an issue yesterday with the log configuration of several servers. This issue has been corrected and statistics should return to normal beginning with today’s report. Unfortunately, active daily user statistics since December 16, 2011 are approximately 33% lower than they should be and cannot be corrected due to lost logs.
More Details
The last few weeks, Mozilla IT has addressed numerous stability issues with AMO and VAMO (versioncheck.addons.mozilla.org, which handles add-on update pings) and made significant changes to the network configuration to keep AMO up and running. One of those changes was the addition of a caching proxy in our Amsterdam datacenter, which was brought online on December 16 to receive 1/3 of the update ping traffic.
Unfortunately, these new servers were not configured to send their logs to our metrics processing tools, so those pings were not counted and the logs are no longer available. We were not immediately sure of the scope of the problem due to both the ongoing stability issues and expected decline in usage during the holidays and fluctuation in usage after a release. The log configuration was corrected yesterday afternoon and Metrics is validating today whether the logs are being processed correctly going forward.
Mozilla IT and Metrics are working on automated ways to prevent this from happening in the future as we continue to deploy AMO around the world. We know that many developers depend on these statistics, and apologize for the data lost over the past 2.5 weeks.
Martha the Livid Watermelon
wrote on