Comparing Different Types of Broken Web Sites

ehergenrader

6

In Part I of our Broken Web Site Reporter analysis here, we looked at some very general data patterns concerning usage of our Report Broken Web Site tool in the Firefox Help menu. Here I will attempt to dive a little further into users’ responses, examining the Problem Type category from the data. When the user reports a broken website, he/she is given nine different problems to choose from:

Picture 1

These categories help us determine what pain points Firefox users experience most often while surfing the web. Let’s take a look at the distribution of of these since the reporter was launched three years ago:

AvgReportsByProb

A first observation — Disability Access has consistently been our top reported website problem. Browsing the comments associated with this category finds many of these reports refer to “Server not found” or “Connection Timed Out” errors. The Disability Access category is meant for websites incompatible with some sort of physical disability, like blindness. It seems from these comments that the name of this category may be confusing or might overlap with some of our other categories.

Taking a look at the Appearance Wrong category, we can see that it has slowly descended from the top reported problem to the fourth from the top. Along with the work done in Firefox 3.0 and 3.5 to display graphics correctly, this decrease has followed a strong increase in the Printed Output is Wrong category introduced in 3.0. It is possible that this newer category is cannibalizing reports from the Appearance Wrong problem type due to their similar connotations. If the printed output in a website is wrong, many users may choose the Appearance Wrong option and vice versa.

Over this time period we can see a relatively linear increase in the Can’t Log In category. Whereas most problem types saw slower growth after the 3.0 launch (as evidenced by their increased flatness), Can’t Log In looks to be growing at about the same rate. It remains to be seen whether or not this trend will continue with 3.5.

What conclusions can we glean from the problem type trends? The Can’t Log In category seems to be the most worrisome. It has been growing steadily since Firefox 2.0, and there is little reason to believe users are confusing Can’t Log In with any of the other categories. Taking a look at specific user comments yields the same results – 35% of en-US users had the word “log in” or a variation in their report. We now know our naming scheme for these reporter problems is flawed; Disability Access sees far too many problems reported, and Appearance Wrong and Printed Output is Wrong are probably feeding off each others’ reports.

Next up in Part III we’ll be looking at the reporter data by locale. Stay tuned!

6 responses

Post a comment

  1. Dan wrote on :

    Sounds like you guys need to disable the Reporter for error pages. None of the problem categories are relevant for pages that don’t load (except maybe “Other”).

    Perhaps the dialog can be made different for error pages, containing text such as “In most cases the error you are seeing indicates a problem with the website and not Firefox. Please ensure the website loads in another browser before submitting a report.” And of course you’d remove the categories since there’s really only one category of problem… the website isn’t loading.

  2. Justin Dolske wrote on ::

    I’m curious what kind of “can’t log in” problems are bring reported… Are these password manager problems, or problems with the site?

  3. ehergenrader wrote on :

    Justin:

    It is difficult to tell, since most of the description fields just say things along the lines of “can’t log in to facebook” or “log in taking way too long”.

    That being said, I’d hypothesize most were site problems (the phrase “password manager” was only mentioned 12 times in the roughly 36,000 en-US descriptions (this includes variations of the phrase with different letters capitalized)).

  4. Donnie Berkholz wrote on ::

    Seems to me that the absolute scale on the Y axis is useless. For the comparisons you want to make, you should be using percentages. Otherwise it’s like putting everything on some weird diagonal X axis that slopes at the rate of increase of problem reports.

  5. Kurt (supernova_00) wrote on :

    I also agree that it sounds like the reporter should be disabled for error pages. Is there a bug filed on this already?

  6. Kurt (supernova_00) wrote on :

    Couldn’t find a bug on it so I filed:
    https://bugzilla.mozilla.org/show_bug.cgi?id=507401

Post Your Comment