Mozilla Labs

What do you get when you mix one part automation, one part natural language interpretation, two parts programming by demonstration, and three parts online collaboration? If you stir all of these research areas together and toss in some XUL, you get one of the most innovative extensions for Firefox: CoScripter.

CoScripter was created by a research team at IBM led by Allen Cypher, and it allows you to record your actions on the Web, play them back, and share them with others. For instance, one popular script quickly automates the process of adding your phone number to the national do not call registry:

A video demonstrating how CoScripter works is available on IBM’s alphaWorks site. In the video, they automate the process of searching for houses in Palo Alto. Instead of bookmarking individual static pages on the Web, this is like bookmarking a series of actions, with online sharing and tagging built in.

In addition to being really useful, CoScripter is also very interesting from a research perspective. One of the most innovative aspects of CoScripter is that actions are represented as human readable and editable text. In their CHI 2007 paper they put CoScripter into the context of previous Firefox extensions:

Greasemonkey [1] enables users to make client-side modifications to the appearance and behavior of web pages on their computer. However, creating a Greasemonkey script requires detailed knowledge of JavaScript programming to alter the DOM of the web page.

Chickenfoot [2] eases client-side customization by providing a higher-level API for accessing and manipulating common web page elements, using information in the rendered DOM. For example, the Chickenfoot instruction click(‘search button’) will click a button with the text “search” on it. However, the Chickenfoot interface is still very much a programming interface, in which users write syntactically correct statements in the Chickenfoot programming language.

Unlike Greasemonkey and Chickenfoot, CoScripter does not require users to know how to program. CoScripter expresses commands in natural language, as opposed to a formal scripting syntax. This means that you can literally edit the textual instructions and play the script again, or even drop in instructions written by hand and see if CoScripter is able to execute them. Because CoScripter’s interpreter is extremely flexible, this actually works surprisingly well. They call this approach Sloppy Programming:

…Koala [former name] leverages the sloppy programming approach in the web domain by taking advantage of the fact that most web commands are flat: there is one verb, and one or two arguments. This assumption dramatically simplifies the algorithm, and makes it more robust to extraneous words. It can handle long expressions originally intended for humans.

Some of the code that drives CoScripter is also interesting from an accessibility perspective. Imagine commanding your browser using only your voice, or tabbing through form fields on a Web page and having a screen reader accurately tell you what each element is by analyzing the surrounding text. Since the CoScripter team plans to open source their code, Mozilla’s accessibility team will be looking into leveraging their work.

In the future CoScripter might also impact how we test Firefox. Ray Kiddy recently wrote a post proposing that we allow beta testers to attach a log of their actions to a bug report, instead of having to manually write a list of steps explaining how to recreate the issue. Ray notes that in addition to helping our testers quickly communicate the steps to recreate a bug, these scripts could also eventually be used for automated testing.

Quick note about phishing: since scripts are shared between users, be careful what you run. Hopefully the social nature of CoScripter will result in the community quickly flagging and removing any malicious scripts that get submitted.

To everyone at IBM that worked on building CoScripter, congratulations on setting a new bar for the state of the art in Web browser automation.

One of the cool new features coming in Firefox 3 is support for a new animated image format. Browsers have supported animated GIFs for more than a decade, but the GIF image format has a number of limitations and is overdue for replacement. The PNG image format is now widely accepted as a superior replacement for static GIF images, but for animated GIFs there has not yet been a clear successor. The new Animated PNG format (APNG) is a simple extension to PNG, making it superior for animations too.

The problem

So, what’s wrong with GIFs? The two most apparent problems are that GIF images can only contain 256 colors, and lack support for partially-transparent pixels. This results in images which often look grainy or rough, and which do not smoothly blend into their background. A clever designer can somewhat mitigate these problems, but the tricks used don’t always work well in different situations — especially if the image is animated. Let’s look at an example of this problem…

Example: the Firefox “throbber”

In Firefox 2, the animation used to indicate that a page is still loading (affectionately known as a “throbber“) is an animated GIF. Here’s the actual image used on OS X, placed against some different backgrounds:

.demoimg1 { padding: 1em; background-color: #404040;} .demoimg2 { padding: 1em; background-color: #ebebeb;} .demoimg3 { padding: 1em; background-color: #40ff40;} .demoimg4 { padding: 1em; background-color: #dd0041;} .demoimg5 { padding: 1em; background-image: url(http://people.mozilla.com/~dolske/apng/checker.png);}

You can see that when used on a light gray background (OS X’s default color), the throbber looks just fine. But against any other background, it looks terrible. Why? To antialias the image, in order to make it look smooth, pixels must be blended to a fixed color (the expected background). If partial transparency was supported, the antialiased pixels could be made appropriately transparent, and the animation would look fine against any background.

Here’s the throbber, now rendered as an APNG: [If you’re viewing this page with Gran Paradiso, the development version of Firefox 3, you’ll see the images below as they, uhh, throb. If you’re not, you should just see the static first frame — APNG is backwards compatible with existing PNG viewers.]

The APNG version clearly looks better against different backgrounds. And that’s useful to have in practice, as it allows Firefox themes and extensions (like Chromatabs 🙂 to change the browser’s chrome while reusing the existing throbber, and not end up with an ugly mess.

A chicken and egg…

A spiffy new animation format is nice, but without an authoring tool no one will use it. But who’s going build such a tool before anyone is using the format? Chicken, meet egg.

Luckily, here at Mozilla Labs we have the technology to build both the chicken and the egg! The Mozilla platform can encode as well as decode APNG images, runs on multiple platforms, and has familiar environment for dealing with UI/scripting/graphics. So, yadda yadda, I wrote a Firefox extension to make creating APNGs easy. It’s called “APNG Edit.” Not the most creative name, but it’ll do. Here’s what it looks like:

My goal was to create a simple editor; the primary workflow being to load up a series of images (one for each frame), adjust the various settings for each frame, and then encode and save the finished animation as an APNG file.

The red dinosaur head shown in the screenshot above is “Chompy,” who serves as an amusing interstitial while Bugzilla processes a user’s search query. Chompy looks a little rough as an animated GIF; problems similar to the GIF throbber above are visible, and it also shows some dithering artifacts.

I obtained the individual frames as PNGs from Klowner, the Mozilla community member who originally rendered Chompy, and was able to quickly load them into APNG Edit and save the resulting animation. Here’s Chompy as a high-quality APNG:

…with a side of bacon

Oh, and “one more thing”!

I quickly realized that while a simple editor to assemble frames was a good first step, a Mozilla-based editor could be a lot more powerful with some minimal effort. Why not take advantage of Javascript and <canvas>, so that animations can be dynamically created and modified? Instead of drawing each frame in Photoshop or Gimp (over and over, as you fine-tune how you want the animation to look), it’s far easier to script the drawing tools provided by <canvas> so that the animation can be generated automatically with the parameters you specify.

For example, to make the OS X throbber, I wrote a script to draw lines with the appropriate shape, color, and rotation for each frame. I was curious what a larger version might look like, so I basically just changed the X and Y size specified in my script, and ran it again:

Here’s an existing <canvas> demo, which I easily converted into a script for generating an APNG:

The first version of the clock animation resulted in a rather large file (about 450K) because each frame was a complete rendering. I wrote another script to optimize each frame by only encoding interframe differences, which knocked the size down to 79K. I could also make easy adjustments to the dimensions or frame count, by simply adjusting the script parameters and re-rendering the animation .

Try it, extend it

Try out the editor for yourself. You’ll need:

• Gran Paradiso Alpha 7 (or newer). APNG Edit does not work on Firefox 2.
• The APNG Edit extension from the Mozilla Addons site.
• (Optional) Some sample scripts and images to experiment with.

There are still some rough spots in the editor which need improved, and there are a lot of interesting possibilities for new features. Unfortunately I anticipate most of my time will be spent working on Firefox 3; however I would welcome patches and scripts from anyone interested. Here’s some ideas for future directions:

• Add the ability to edit frames in an existing APNG or animated GIF (needs changes to imgIDecoder)
• Reorder frames by drag’n’drop
• Save APNGs as color mapped 8-, 4-, or 2-bit images (for decreased file size)
• Scripts to perform more kinds of frame manipulations.
• Sandboxing of scripts to prevent malicious code

• Cropping frames
• Scaling frames
• Animated text with the new <canvas> drawString() API

Gratuitous spinning logo

WIMPy and the Terminal

The history of user interfaces can be very briefly summarized into two distinct eras: the command line, followed by the graphical user interface. Interactions on the command line are very fast, but the set of possible commands is not discoverable. GUIs are essentially the opposite, on both issues. Interactions with graphical user interfaces are slower, but possible commands are given visual affordances, and icons attempt to convey possible commands through metaphors.

The GUI is largely considered superior to command line interfaces that predated it, but that isn’t entirely true. For instance, while I was in college a majority of students preferred Pine (screenshot) over graphical email clients like Outlook. A group of students in a human computer interaction class I was in did an in-depth analysis of the usability of each application. They found that across a wide variety of metrics, like simplicity, system response time, and (most critically) overall time on task, Pine knocked Outlook’s toolbar-customizing-dialog-poping-drag-and-drop socks off.

Tradeoffs

Instead of trying to conclude which is superior, a GUI or a keyboard-based interface, it is important to note the specific tradeoffs each interface currently makes in terms of the bandwidth of output, and bandwidth of input.

Modern graphical user interfaces are clearly higher bandwidth than text-based command line interfaces in terms of output, but consider the bandwidth of input:

Standard GUIs, with their drop down menus, check buttons, and tree-lists just cannot compare to the range of options that a text interface gives effortlessly. In just five alphanumeric characters, you can choose one out of 100,000,000 possible sequences. And choosing any one sequence is just as fast as any other sequence (typing five characters takes roughly 1 second). I challenge you to come up with a non text-based interface that can do as well. (Command Line for the Common Man: The Command Line Comeback)

Graphical user interfaces often provide keyboard shortcuts to serve as accelerators. But these keyboard shortcuts are not interfaces in themselves, but simply serve as hooks into various parts of the GUI. For instance, consider control-D in Firefox, it simply pops up the bookmark creation dialog box, and suddenly the user has to go back to using the mouse (or awkward tabbing) in order to complete their task.

The Best of Both Worlds

Over the last six months, I’ve been thinking a lot about the work of two designers: Nicholas Jitkoff (Blacktree, creators of Quicksilver) and Aza Raskin (Humanized, creators of Enso). Both have designed user interfaces that exist in the riven between command line interfaces and graphical user interfaces. And both of these applications are a joy to use.

Unfortunately these types of hybrid keyboard/GUI user interfaces have gone largely unexplored by interaction designers. Aside from feed and label navigation in Google Reader, I don’t know of too many other applications that are currently leveraging these types of incredibly streamlined graphical interfaces, designed solely for keyboard input.

How Firefox Could Potentially Leverage Graphical Keyboard User Interfaces

Here are some ideas I’ve had about how several different Firefox features could be designed using a graphical keyboard user interface. Please note that these are all only conceptual mockups, and we currently have no official plans to implement these features for Firefox 3 (although, we may at some point release a prototype extension through Mozilla Labs). If you are an extension developer and are interested in contributing to a project like this, please email me or leave a note in the comments.

All of these mockups show interfaces that are entirely keyboard driven. A keyboard shortcut launches the UI, and the UI is later dismissed by either selecting an item using the arrow keys and hitting enter, or by hitting escape. These interfaces are all modal, and when invoked they occupy large amounts of space on the screen.

For each of these mockups you can click through for a larger version.

Searching the Web (control+k, or alt-alt)

Of all of these mockups, I think keyboard-based Web search would be the most useful. This mockup also features some favicon upscaling code I wrote for another Mozilla Labs project.

In addition to Web search, the “Bookmarks and History” search will likely be more efficient than the current WIMPy ways of accessing bookmarks in Firefox:

The move back to language started with web search engines in general, with Google placing the capstone when its name became the house-hold verb for “typing to find what you want”. In fact, Googling is almost always faster then wading through my bookmark menu (which says there is something wrong with using menus as a mechanism for accessing bookmarks). (Command Line for the Common Man: The Command Line Comeback)

Switching Tabs (control+tab)

Navigating Recent History

Tagging Pages

Acting on Microformatted Content

Conclusion

-Just because the command line predated the graphical user interface doesn’t mean interfaces based on windows, icons, menus and pointers are always superior to interfaces based around using the keyboard for input.

-Designing interfaces based solely around the mouse and standard GUI widgets, and adding keyboard accelerators as an afterthought, does not always result in creating the most effective and streamlined user interfaces for advanced users.

-Interaction designers should consider designing keyboard-based graphical user interfaces, to simultaneously take advantage of both high bandwidth input, and high bandwidth output.

What is Joey?

When you browse the Web, there is a great deal of important content that becomes inaccessible as soon as you walk away from your personal computer. Whether it’s the driving directions that you looked up or your favorite sports teams’ latest scores, you lose access to that information as soon as you turn off your computer.

Mobile browsing is intended to solve this problem: just use the browser on your phone to access the content you need. However, as we all know, this isn’t the greatest experience; there is too much data to download, the data is hard to reach, the content isn’t formatted for the phone. Quickly retrieving specific pieces of information using a mobile device needs to be easier.

Project Joey brings the Web content you need most to your mobile phone by allowing you to easily send it to your device. You can quickly mark content that is important to you and have that content always available while using your mobile phone.

The premise is this: you can use Firefox to send text clippings, pictures, videos, RSS content, and Live Bookmarks to your phone through the Joey Server. The Joey Server transcodes and keeps all of the content up-to-date. You can then use your phone’s browser or the Joey application on your phone to view and manage what you have uploaded.

The goal is to investigate the mobile space using a desktop-centric model. Like many other Mozilla projects, there are no plans to productize Joey. We will work with the community to build a compelling and usable tool that people will enjoy using.

Although we are in the early stages of development, much of Joey is usable today. Feel free to go to joey.labs.mozilla.com using Firefox, create an account, and give Joey a try. The Joey application for mobile devices isn’t complete yet, but in the meantime you can access information on the Joey server from your mobile device using any type of mobile browser.

Developer Information

Want to help out? Great! You can start by reading the documents below, checking out the code, and looking over our to-do list. If you’re developing, please get on IRC and join #joey.

View the source code

Important Documents

Communication Protocol
Database Design

Discuss Joey

In addition to commenting below, you can also discuss joey on the labs forums.

Just a few years ago, one of the most popular uses of the internet was to send jokes, cute pictures, and news stories to friends and family (social networks) via email. Fast forward to today, and you’ll find that not a lot has changed except the medium. Now one of the most common uses of instant messenger services and social networking sites is to send people links to jokes, cute pictures and interesting news stories.

This behavior isn’t at all surprising – it’s friends sharing experiences, the very backbone of social interactions and friendships. It’s also unsurprising that most “Web 2.0” services have a feature that makes it easy for you to build a social network so that you can share things more easily, or subscribe to a friend’s activity as a way of keeping in touch.

What is surprising, however, is how little of this type of functionality has made it into today’s web browsers. The result is that when people think of tools for social interaction, email and instant messenger are at the top of their list, not web browsers.

Enter “The Coop”, a Mozilla Labs project to experiment with adding social tools to the web browser. We want to create a fun and easy way to share links with your friends, and to browse the set of links that friends have shared with you. We also want to make it easy to “subscribe” to a friend in order to make it easy to keep track of the pictures, movies, blog posts and status information that they might be posting on a variety of services. There’s a project page that describes The Coop in a bit more detail, and also has some mockups of how it might look (my favorite is the idea for a view that shows a stream of recently shared material.)

Myk Melez has also made an initial prototype available. (Note that you’ll have to allow add-ons to be installed from labs.mozilla.com in order to install The Coop.) This first build uses Facebook’s “Share” feature as the data transport layer for now, and allows you to share web content by dragging it onto your friend’s picture. As the project page indicates, we’re thinking of several different data transport mechanisms, as well as how we want to expose various interactions. This prototype really helps to get a feeling of what The Coop might become over time.

If you’re interested in an easier way to share your cool online experiences with your friends, we encourage you to check out the project page and the prototype (one cool feature of the new AMO sandbox is that you can browse the code), and then come join us in the forums and help us iterate on the design and code.

Update! We have made The Coop available at a new location here and have updated the original link above. Note that you will have to allow add-ons to be installed from labs.mozilla.com in order to install this prototype.