When I began my analysis of the Broken Site Reporter (seen above) in Firefox (first post can be read here and second here), I imagined a series of 4 or 5 posts with a concluding post on changes I thought would be beneficial to the tool. For now, the end is near for the Reporter series. Why did I cut the analysis short? In my mind, the Broken Site Reporter in its current form leads to too many data issues to draw a whole lot more meaningful analysis from it. The subsequent changes I propose in this post should alleviate these concerns, and after getting fresh data we should be able to breath life back into the series.

So what are the problems we’ve seen data-wise with the Reporter? The first issue that comes to mind is category-confusion. By this I mean certain categories are underrepresented/overrepresented because users are getting confused about where their individual problem fits with respect to the categories we let them choose from. Here are the category types:

Three categories that stand out are Behavior wrong, Appearance wrong, and Printed output is wrong. Both Appearance wrong and Printed output is wrong seem to be subsets of Behavior wrong (for that matter, most reported problems technically belong to subsets of Behavior wrong). It is also unclear what meaning is assigned to Printed output is wrong — is it output printed to the screen (in which case the category is dangerously close to Appearance wrong) or is it output physically printed on paper? In this case, some sort of example next to the category could help my confusion.

Another problem is that Other content missing borders close enough to Other problem that the latter is going to cannibalize reports meant for the former.

With these issues at hand I have come up with a very rough sketch of changes I’d like to see made to the Reporter (Keep in mind that I know absolutely nothing about UI — these pictures are more intended to outline my ideas in a concise way than to provide any sort of finalized draft of what the Reporter should look like; criticism is welcome).

The biggest change in this new Reporter is the narrowing of the problem types. I have drilled them down to five distinct problems (Behavior wrong, Disability access, Printed output is wrong, and Other content missing have all been eliminated). In this way, much of the old Behavior wrong and Printed output is wrong responses can be absorbed into the more concise Appearance Wrong category; reports from these categories that do not get redirected to Appearance Wrong will hopefully be sent to Other Problem. The question mark next to the categories is meant to be a link to a site with descriptions of each problem type and examples of reports and their corresponding appropriate category.

Disability access represents a major hurdle for data analysis in the current Reporter implementation. The category is meant for human-disabilities, but most of the reports it receives are from users’ connections timing out or “Server not Found” errors. These are not the types of errors the Reporter is built for, and they end up skewing our data. In light of this, I am proposing that the Reporter detect these errors when present, and when a user tries to report a broken website, give the user the option of reporting the error in a way like this:

With an error checking page such as this we can both clean up our data and get new useful information about the state of different websites (which sites are down at what time, how often certain sites are reported as down, etc.).

With modifications to the Reporter like these we can get out much cleaner data that will help us better respond to problems in Firefox. Hopefully we’ll see some changes to the tool soon so we can get more actionable data!