Alex FaaborgUser Experience Design at Mozilla

… it might sound counter intuitive for a project so intently focused on usability and creating a mainstream browser, but in all seriousness I believe that Mozilla needs to (caveat: “for the most part”) stop talking about users.

Last week I attended a workshop about usability in open source projects. This gave me a chance to meet with designers from a number of other projects and organizations including Ubuntu, KDE, RedHat and OpenOffice.org, as well as members from the HCI research community. We each wrote a position paper prior to attending, and the position papers written by myself and Andy Ko from the University of Washington ended up complimenting each other well. Andy articulates the problem, and I provide a solution.

First, the problem:

100 contentious Firefox bug reports were analyzed for distinct uses of the word “user.” The results show that developers use authoritative words (such as allow, educate, and require) to describe what software does for users. Most statements involved confident speculation about what users need, expect and do, whereas a minority of statements demanded evidence for such unsubstantiated claims. The results also show that when describing users, developers describe them in general terms, rather than referring to particular uses cases or user populations. These results suggest that, at least in the broader Firefox developer community, developers rely largely on stereotype and instinct to understanding user needs. [From How do Open Source Developers Talk About Users]

Now this sounds pretty bad, we are confidently proclaiming speculative statements that lack any form of evidence. It’s perhaps not quite as dire as it initially sounds, since while we are falling back to stereotype (generally “users who are not developers”) and instinct (“we should make this easier”) at least the community is debating changes with some notion of users in mind. However, when you invoke a hypothetical user for use in debate (often someone’s mother), you can pretty much project whatever you want onto that hypothetical user in an attempt to win an argument. Stereotype and instinct make interface design seem arbitrary and subjective, they lack precision.

Here’s a proposed solution to this problem which I co-authored with Daniel Schwartz (you can also read it in the standardized academic conference format). Instead of talking about users, Mozilla should instead focus debate on core user experience principles.

Using a Distributed Heuristic Evaluation to Improve the Usability of Open Source Software

Building tools to enable a large scale distributed Heuristic Evaluation of usability issues can potentially reshape how open source communities view usability, and educate a new generation of user experience designers. We are exploring adding the ability to perform a distributed Heuristic Evaluation into the Bugzilla instance used to develop Firefox. Ideally this approach will build more of a culture around HCI principles, and will create a framework and vocabulary that will cross pollinate to other open source projects.

Introduction

When contemplating how to increase the influence of user experience design in open source communities, a common approach is to attempt to “increase the involvement and visibility of UX professionals” [1]. However, this paper asks a different question: how can we convert current developers in open source projects to have a skill set equivalent to what academia or a corporation would consider to be formally trained user experience professional? The approach we propose consists of embedding HCI concepts and practices directly into the tools that control an open source community’s work flow.

Beyond controlling an open source community’s process and work flow, tools also indirectly shape the community’s values and ideals. This is important, because if social currency in the community is inherently linked to one’s ability to make the software better, the concept of better must be expanded to encompasses “easier to use.”

Quantitatively Measuring Usability

Measurements like the time it takes an application to load, the amount of memory used, or load on the cpu are all trivial to calculate, and wonderfully quantitative. One of the reasons open source communities tend to discount usability (both in practice and in artifacts like the severity descriptions in Bugzilla), is an inaccurate view that usability is an amorphous, and subjective thing that simply can’t be scientifically quantified and measured. However, measuring an application’s usability is an area where previous HCI research can make a strong and very significant contribution to open source development.

The usability inspection technique of Heuristic Evaluation, which was introduced by Jakob Nielsen [2,3,4] has emerged as one of the most common ways for professional user experience designers to evaluate the usability of a software application. Heuristic Evaluations are extremely useful because they formally quantify the usability of a software application against a set of well defined and irrefutable principles. Usability violations can be quantified individually: either an interface supports undo, or it does not, either an interface is internally consistent, or it is not, etc. Usability violations can also be quantified in aggregate: the software application currently has 731 known usability issues. Additionally, by building the tracking system on a set of agreed upon principles, much of the debate on the level of “there is no right or wrong with UI / every user is entitled to their personal opinion / all that matters is the ability to customize” which is currently found in open source software development communities may be significantly reduced. Usability heuristics will help ground these debates, just as currently no one in an open source community argues in favor of data loss, or in favor of crashing.

Injecting HCI Principles into Bugzilla

Adapting an open source community’s bug tracker to capture usability issues defined by a set of specific heuristics can reshape the way developers think about usability. Just as open source development communities currently have a shared vocabulary to describe good and bad with concepts such as performance, data loss, and crashing, usability heuristics can introduce additional concepts, like consistency, jargon, and feedback. All of these concepts, covering both the underlying implementation as well as the user interface, can now have an equal potential to impact the software application at any level of severity, from trivial to critical.

Modifying a bug tracking system to track a Heuristic Evaluation of software is reasonably straightforward. Each issue needs to be able to be associated with the specific usability heuristic being violated (for example: “using the term POSTDATA in a dialog is technical jargon“). We plan to utilize Bugzilla’s keyword functionality, similar to how current bugs can be flagged as violating implementation level heuristics, like data loss. Since the evaluations will be performed by contributors who likely will not have any additional interface design training, it is important that each heuristic is very clearly defined with specific examples and detailed explanations. Additionally, allowing contributors to view all of the bugs in the software marked as the same type of issue, both current and resolved, serves as an effective way for them to further learn about the heuristic.

We are now working to embedded to functionality needed for a distributed Heuristic Evaluation into the Bugzilla instance used to develop Firefox and Thunderbird. These specific modifications may spread to a variety of other open source projects as Bugzilla is currently used by communities including the Linux Kernel, Gnome, KDE, Apache, Open Office and Eclipse [5]. Ideally embedding HCI principles into development tools will also embed the ideals into the community. Similar to other forms of bugs, there will be a social incentive for contributors to locate and classify violations, and there will be a social incentive for other contributors to resolve them. As open source contributors travel between different communities and projects, the usability heuristics will also see a similar cross pollination between open source communities. A shared vocabulary will emerge across open source projects, allowing for clearer communication and debate.

Side Effects of Distributed Heuristic Evaluation

Today the process of Heuristic Evaluation is normally completed in corporations and academia by a small number of designers, who are extremely well practiced at identifying usability issues. However, it is worth noting two important aspects of the Heuristic Evaluation method from when it was first introduced:

Education – First, the method of Heuristic Evaluation has its roots not in the functional purpose of evaluating usability, but rather in the even more basic purpose of teaching usability. We see this in Nielsen’s 1989 SIGCHI bulletin: Teaching User Interface Design Based on Usability Engineering [2] that Heuristic Evaluation was introduced as part of the curriculum for a masters degree in Computer Science. This is still true today: the road to becoming a good user experience designer begins with mastering the identification of well defined heuristics.

Power in Numbers – The second important aspect of Heuristic Evaluations is that it was quickly found that the number of evaluators played a major role in how successful it was. Nielsen wrote in 1990 that “evaluators were mostly quite bad at doing such heuristic evaluations… they only found between 20 and 51% of the usability problems in the interfaces they evaluated. On the other hand, we could aggregate the evaluations several evaluators to a single evaluation and such aggregates do rather well” [3]. For large open source projects, Bugzilla instances often have thousands to hundreds of thousands of users.

Conclusion

In open source software development, the educational and distributed aspects of a Heuristic Evaluation are critically important. While the majority of open source projects currently lack user interface designers capable of performing a perfect Heuristic Evaluation in isolation, that’s irrelevant. The collaborative nature of open source projects allows for a group of contributors to effectively compete with a formerly trained user experience professional by aggregating their abilities. And similar to all of the other ways in which people contribute to open source projects, there is a mutually advantageous feedback loop: the more effort a contributor puts into improving the software, the more they are able to increase their own skill set. Hours spent performing heuristic evaluations and brainstorming ways to address usability issues will allow a new generation of user experience designers to emerge in open source communities, just as rock star software develops are currently forged there.

References

1. Schwartz, D. and Gunn, A. 2009. Integrating user experience into free/libre open source software: CHI 2009 special interest group. CHI EA ’09. ACM, New York, NY, 2739-2742.
2. Nielsen, J. and Molich, R. 1989. Teaching user interface design based on usability engineering. SIGCHI Bull. 21, 1 (Aug. 1989), 45-48.
3. Nielsen, J. and Molich, R. 1990. Heuristic evaluation of user interfaces. CHI ’90. ACM, New York, NY, 249-256.
4. Nielsen, J. 1994. Enhancing the explanatory power of usability heuristics. CHI ’94. ACM, New York, NY, 152-158.
5. Bugzilla Installation List, http://www.bugzilla.org/installation-list

Next Steps

Getting this set up for bugzilla.mozilla.org is now being covered in bug 561262. The initial set includes 17 core principles, which we may expand or modify over time as we apply these to all of the user experience bugs that we are currently tracking. The Firefox UX team will be pushing on these usability heuristics pretty heavily during the development of Firefox 4. Also some other significant open source projects are looking into using a similar approach for their bug databases, so hopefully this shared vocabulary for usability will begin to cross pollinate across the open source community.

Business Week recently wrote a story about how Mozilla Labs Explores Open Source Design. Overall it was a great article, although it may have slightly misrepresented the role of voting in the design challenge process. It was probably the phrase “submissions are put to a public vote” that triggered Nielsen’s response:

“There are plenty of good ideas, but they don’t work well together with the real world,” says Jakob Nielsen, principal at the Nielsen Norman Group in Fremont, Calif. Open source encourages the addition of new solutions and ideas. In design, says Nielsen, “the brilliant idea can be the one unifying idea that can take away 10 other ideas.”

This quote has been stuck in my head for awhile, not because I specifically disagree but because it so accurately describes the paradox of doing design work in a transparent, collaborative, and open source environment. There is certainly no shortage of great ideas coming from our community, in large part thanks to the success of the Mozilla Labs Design Challenges. But as Nielsen points out, design isn’t just about brainstorming, it’s about unifying things, taking things away, and being able to say no. And this is where I think design in open source is often unfairly misconstrued to be based on voting and consensus building, effectively “design by committee.” People familiar with open source projects know that there is in fact centralized control both in terms of implementation and in terms of design.

What makes open source communities different is that contributors gain influence in a pure meritocracy. For instance, the design of the Firefox icon came out of the community, from a team of people who would later help to guide all of Mozilla’s visual identity. Concepts submitted to the recent Mozilla Design Challenge are impacting both the design of Firefox (a home tab that changes based on the user’s geolocation) and Chrome OS (proposal, submission). Our icons on Linux are created in the Tango style, because the Tango team has earned a tremendous amount of respect. Across both visual and interactive design we see a wide range of very successful contributions.

We also see a lot being turned down, patches failing review, bugs being marked wontfix, and the extremely common phrase “that would make a good extension” (translation: that is not going to make it into the product for mainstream users).

So I agree with Nielsen that “Open source encourages the addition of new solutions and ideas” and that “[designer's create] the one unifying idea that can take away 10 other ideas.” Where I disagree is the implication that open source projects can not successfully do both.

As usual there was a tremendous amount of research exploring the interface of Web browsers (and usually implemented on top of Firefox) at this year’s CHI conference in Atlanta. Here is a quick overview of some of the work. Of the 10 papers studying or modifying Firefox, the main theme this year was content sharing and rich web history, which about half of the papers explored.

1) Sharing and History

Eyebrowse: Real-Time Web Activity Sharing and Visualization
Max Van Kleek, Christina Xu, Brennan Moore, David R. Karger
In this paper,we explore the potential for letting users automatically track and selectively publish their web
browsing activities in real time on the Web. We developed a system, Eyebrowse, with three goals: first, to provide a means for individuals to better understand how they spend time on the web through visualizations and statistics; secondly, to foster social discovery and awareness through real-time web activity sharing;and finally, to build a large public corpus of web browsing trails using this method. We gathered user impressions of Eyebrowse, including perceived usefulness, feelings of self-exposure,and privacy concerns, for ascertaining ways to improve the system.
Web Site
Video
Paper

Here’s What I Did: Sharing and Reusing Web Activity with ActionShot
Ian Li, Jeffrey Nichols, Tessa Lau, Clemens Drews, Allen Cypher
ActionShot is an integrated web browser tool that creates a fine-grained history of users’ browsing activities by continually recording their browsing actions at the level of interactions, such as button clicks and entries into form fields. ActionShot provides interfaces to facilitate browsing and searching through this history, sharing portions of the history through established social networking tools such as Facebook, and creating scripts that can be used to repeat previous interactions at a later time. ActionShot can also create short textual summaries for sequences of interactions. In this paper, we describe the ActionShot and our initial explorations of the tool through field deployments within our organization and a lab study. Overall, we found that ActionShot’s history features provide value beyond typical browser history interfaces. share interesting web pages that they found. However, these web sites only allow people to share the URLs of individual pages. If people want to share what they did on a web site, they have to write it down manually, which can be so tedious that they forego sharing the information. Social scripting services such as CoScripter [8] allow users to record and share interactions with websites, but these tools require forethought and planning to enable recording at the right time to capture a reusable script. Moreover, CoScripter’s one-to-all sharing model was found to deter many users [8], who asked for finer grained control over with whom they shared their scripts. An enhanced browser history could solve these problems, letting users easily grab sequences and share them with the desired audience.
Web Site
Paper

A Task-focused Approach to Support Sharing and Interruption Recovery in Web Browsers
Mohan Raj Rajamanickam, Billy Lam, Russell MacKenzie, Tao Su
Over the last two decades a vast number of services have moved online, and many new services have been created. Previous work shows that many users are overloaded by the number of webpages they use simultaneously. We introduce TabFour, a prototype web browser which integrates three features that address the design requirements identified in an initial design study. Webpages can be grouped into tasks, providing a unified target for resumption after an interruption. Tasks and pages can be annotated, supporting resumption after longer intervals. Finally, tasks can be shared through a simple yet novel web-service, allowing users to share groups of webpages more easily than with existing tools.
Paper

Enhancing Directed Content Sharing on the Web
Michael S. Bernstein, Adam Marcus, David R. Karger, and Robert C. Miller
To find interesting, personally relevant web content, people rely on friends and colleagues to pass links along as they encounter them. In this paper, we study and augment link-sharing via e-mail, the most popular means of sharing web content today. Armed with survey data indicating that active sharers of novel web content are often those that actively seek it out, we developed FeedMe, a plug-in for Google Reader that makes directed sharing of content a more salient part of the user experience. FeedMe recommends friends who may be interested in seeing content that the user is viewing, provides information on what the recipient has seen and how many emails they have received recently, and gives recipients the opportunity to provide lightweight feedback when they appreciate shared content. FeedMe introduces a novel design space within mixed-initiative social recommenders: friends who know the user voluntarily vet the material on the user’s behalf. We performed a two-week field experiment (N=60) and found that FeedMe made it easier and more enjoyable to share content that recipients appreciated and would not have found otherwise.
Web Site
Paper

Enabling Cross-Device Interaction With Web History
Timothy Sohn, Koichi Mori, Vidya Setlur
Internet-enabled personal devices are growing in number. As people own and use more devices, sharing information between devices becomes increasingly important. Web browsing is one of the most common tasks, thus sharing web history is a first step in supporting cross-device interaction. Current methods of sharing web history involve manual, cumbersome methods. This paper explores a system to automatically synchronize web information among a user’s personal devices, and optimize the interface to support mobile users. We describe a system that enables users to quickly find directions on their mobile phone based on past web searches, and seamlessly share favorite web pages between their personal devices.
Paper

2) Tabbed Browsing and Tasks

A Study of Tabbed Browsing Among Mozilla Firefox Users
Patrick Dubroy, Ravin Balakrishnan
We present a study which investigated how and why users of Mozilla Firefox use multiple tabs and windows during web browsing. The detailed web browsing usage of 21 participants was logged over a period of 13 to 21 days each, and was supplemented by qualitative data from diary entries and interviews. Through an examination of several measures of their tab usage, we show that our participants had a strong preference for the use of tabs rather than multiple windows. We report the reasons they cited for using tabs, and the advantages over multiple windows. We identify several common tab usage patterns which browsers could explicitly support. Finally, we look at how tab usage affects web page revisitation. Most of our participants switched tabs more often than they used the back button, making tab switching the second most important navigation mechanism in the browser, after link clicking.
Blog Post with the presentation
Paper

Multitasking Bar: Prototype and Evaluation of Introducing the Task Concept into a Browser
Qing Wang, Huiyou Chang
This paper clarifies two common patterns of multitasking on the Web, namely Multiple Tasks (MT) and Multiple Session Task (MST). To support both of these, the task concept needs to be introduced into a browser. An online pilot survey has revealed which attributes of the task concept are most significant to Web users and as a result a simple prototype, the Multitasking Bar (MB), is proposed based on these findings. The MB copes with the multitasking needs of both MT and MST in the browser by providing functions for task related Web page management and task schedule management. A two-session controlled experiment has been conducted to evaluate the MB and to compare user performance and experience when multitasking on the Web with and without support for MT and MST. Results show that support for both MST and MT significantly improves user task performance efficiency and greatly enhances the user experience when multitasking on the Web.
Paper

3) Improving the Interface

Enhancing Web Page Readability for Non-native Readers
Chen-Hsiang Yu, Robert C. Miller
Readers face many obstacles on today’s Web, including distracting content competing for the user’s attention and other factors interfering with comfortable reading. On today’s primarily English-language Web, non-native readers encounter even more problems, even if they have some fluency in English. In this paper, we focus on the presentation of content and propose a new transformation method, Jenga Format, to enhance web page readability. To evaluate the Jenga Format, we conducted a user study on 30 Asian users with moderate English fluency and the results indicated that the proposed transformation method improved reading comprehension without negatively affecting reading speed. We also describe Froggy, a Firefox extension which implements the Jenga format.
Web Site
Paper

Cookie Confusion: Do Browser Interfaces Undermine Understanding?
Aleecia M. McDonald
We performed a series of in-depth qualitative interviews with 14 subjects recruited to discuss Internet advertising. Participants held a wide range of views ranging from enthusiasm about ads that inform them of new products, to resignation that ads are “a fact of life,” to resentment of ads that they find “insulting.” We discovered that many participants have a poor understanding of how Internet advertising works, do not understand cookies, and mistakenly believe there are legal protections barring companies from sharing information they collect online. We found that participants have substantial confusion about the results of the actions they take within their browsers, and do not understand the technology they work with now. The user interface for cookie management in popular browsers may be contributing to confusion.
Paper

Remote Web Browsing via the Phone with TeleWeb
Yevgen Borodin
TeleWeb is an assistive voice-enabled application empowering users to remotely access the Web through the most ubiquitous device – the phone. The uniqueness of the technology is that it enables users to gain access to information from almost anywhere via a plain, old-fashioned telephone. TeleWeb users will be able to call their own personal numbers, authenticate themselves, and then use speech and phone key-pad to remotely browse the Web on their own PCs. TeleWeb may especially appeal to people with vision loss, as well as older adults who may find the phone interface to be more familiar and easier to use. In this paper, I describe the TeleWeb approach and the interface.
Video
Paper

Conclusion

It was really great to see so many brilliant people thinking about Firefox and the Web in general. In fact there were so many papers covering topics related to Firefox that I often had to jump between session running concurrently in order to catch everything. It’s also exciting to see that even outside of the scope of Mozilla Labs there is a really broad range of research going on in the academic community exploring the potential future of the Web.

Last night Boriss wrote a great post about the benefits of the contextual design process. Aspects of the contextual design process like the inquiry, work modeling and environment design are all incredibly important skills for a UX designer to have. However, I couldn’t disagree more with the premise that this process should have been applied by Lead Ubuntu designer Ivanka Majic in the design of the window manager.

Limitations of Contextual Design

Contextual design makes a lot of sense when you are considering a product for a rather specific use case. For instance, an application that helps doctors track the medical history of their patients throughout the day, or data visualizations for an investment banker. In these situations there is a lot to discover about the users, their tasks, their underlying goals, their environment, the situations where they fail, the situations where the succeed, etc. Basically, there is loads of context, and potentially a lot of context that hasn’t previously been considered in the horrible interface they might currently have.

So contextual design is really the most effective when designing to a specific audience. And this is also it’s limitation: applying it in situations where there is no specific audience is simply going through the motions and structure of a formal design process for no reason. Who are Ubuntu’s users? Humans. What do they do with their window manager, they want to manage windows.

Limitations of Personas

A similar design process is the use of Personas (not the lightweight Firefox theme, but the notion of creating a small set of hypothetical users who embody the attributes of the users you are designing for). Personas are an effective way to make sure that your product isn’t just connecting with users, but with the right kind of users. For instance, if you consider a game console there is really a pretty wide range of users who it could appeal to, from hard core gamers in their 20s shouting obscenities into their headsets to middle aged women doing Yoga on a balance board. The Xbox 360′s design appears to be very directly influenced by the three specific personas that they targeting: Striker the hard core gamer, BeatBuilder the technology trendsetter, and Velocity Girl, the casual gamer and creator of virtual user generated content. And even though they never really delivered on their persona of Velocity Girl, the vision of having that type of user was nonetheless inspired (in a wanting to implement Snow Crash kind of way). Personas help you focus your product on a specific vision when there are an infinite set of possibilities.

Designing a Better Doorknob

But let’s say we are creating the type of thing that nearly everyone on the planet might end up using at some point, like a window manager, or a Web browser, or a doorknob. Understanding the context is pretty trivial, and building personas for the door knob only really makes sense if the interactive design firm is charging by the hour (kidding).

But does this mean that we must fall into the trap of what Aza criticizes as “wishy-wasy design speak” that makes us look like we are pulling the Science of interaction design out of our collective behinds? Not at all. In fact, you can launch an entire career as a top tier usability pundit simply by quantifying how to build a better doorknob.

Designing Better Window Controls

So if getting context about users through the inquiry process and building personas doesn’t help, what aspects of interactive design should we consider when creating window controls? Off the top of my head two come to mind:

Self Describing Visual Language: the controls should visually represent what they are going to do, without the user having access to any external information like a manual, a helpful friend or memory of having used them before. This is an aspect where the the street light metaphor being used by Apple fails. The first time you look at them you realize that they are communicating using only color (and at one point a fourth control was telling the user “Purple!”). If the user ever waits for a tooltip to fall back to literal language your glyph has failed.

The new Ubuntu controls say more to the user than OS X, but they don’t fare particularly well in being self describing either, visually they seem to communicate that one control will move the window upwards, and another control will move the window downwards, sort of like a scroll bar. They don’t communicate that the window will be transformed.  The visual communication used in Windows is considerably more literal, choices include “big box” and “small thing at bottom.”

Natural Mapping: in general controls should be placed in a position relative to the effect that they will have. For instance, on a lot of stove tops if you have four burners you will also have four dials in the exact same configuration. If controls don’t have a natural mapping and the user falls back to point one (visual language) looking for pictograms of which dial relates to which burner, the natural mapping has failed. Ever turn on the wrong light switch, perhaps every time you try to turn on the correct one? Odds are it doesn’t have a natural mapping to the lights in the room.

For window controls you could make an argument that as the window maximizes it’s corners stretch outward, so the most natural mapping for the maximize control is on the outermost edge (regardless of side). I think the reason we don’t see any of the three designs going for this is that close is simply used so much more that it is worthwhile to give it more prominent visual placement when looking in from the side of the window (even if it violates what would be a better natural mapping).

So there is still plenty that the field of interaction design can offer to the design of simple everyday interactions, even if the context is “using the thing” and the persona is “a person living on planet earth.”