The Graphical Keyboard User Interface

WIMPy and the Terminal

Wimpy

Terminal

The history of user interfaces can be very briefly summarized into two distinct eras: the command line, followed by the graphical user interface. Interactions on the command line are very fast, but the set of possible commands is not discoverable. GUIs are essentially the opposite, on both issues. Interactions with graphical user interfaces are slower, but possible commands are given visual affordances, and icons attempt to convey possible commands through metaphors.

The GUI is largely considered superior to command line interfaces that predated it, but that isn’t entirely true. For instance, while I was in college a majority of students preferred Pine (screenshot) over graphical email clients like Outlook. A group of students in a human computer interaction class I was in did an in-depth analysis of the usability of each application. They found that across a wide variety of metrics, like simplicity, system response time, and (most critically) overall time on task, Pine knocked Outlook’s toolbar-customizing-dialog-poping-drag-and-drop socks off.

Tradeoffs

Instead of trying to conclude which is superior, a GUI or a keyboard-based interface, it is important to note the specific tradeoffs each interface currently makes in terms of the bandwidth of output, and bandwidth of input.

Modern graphical user interfaces are clearly higher bandwidth than text-based command line interfaces in terms of output, but consider the bandwidth of input:

Standard GUIs, with their drop down menus, check buttons, and tree-lists just cannot compare to the range of options that a text interface gives effortlessly. In just five alphanumeric characters, you can choose one out of 100,000,000 possible sequences. And choosing any one sequence is just as fast as any other sequence (typing five characters takes roughly 1 second). I challenge you to come up with a non text-based interface that can do as well. (Command Line for the Common Man: The Command Line Comeback)

Tradeoffs

Graphical user interfaces often provide keyboard shortcuts to serve as accelerators. But these keyboard shortcuts are not interfaces in themselves, but simply serve as hooks into various parts of the GUI. For instance, consider control-D in Firefox, it simply pops up the bookmark creation dialog box, and suddenly the user has to go back to using the mouse (or awkward tabbing) in order to complete their task.

The Best of Both Worlds

Over the last six months, I’ve been thinking a lot about the work of two designers: Nicholas Jitkoff (Blacktree, creators of Quicksilver) and Aza Raskin (Humanized, creators of Enso). Both have designed user interfaces that exist in the riven between command line interfaces and graphical user interfaces. And both of these applications are a joy to use.

Quicksilver

Unfortunately these types of hybrid keyboard/GUI user interfaces have gone largely unexplored by interaction designers. Aside from feed and label navigation in Google Reader, I don’t know of too many other applications that are currently leveraging these types of incredibly streamlined graphical interfaces, designed solely for keyboard input.

How Firefox Could Potentially Leverage Graphical Keyboard User Interfaces

Here are some ideas I’ve had about how several different Firefox features could be designed using a graphical keyboard user interface. Please note that these are all only conceptual mockups, and we currently have no official plans to implement these features for Firefox 3 (although, we may at some point release a prototype extension through Mozilla Labs). If you are an extension developer and are interested in contributing to a project like this, please email me or leave a note in the comments.

All of these mockups show interfaces that are entirely keyboard driven. A keyboard shortcut launches the UI, and the UI is later dismissed by either selecting an item using the arrow keys and hitting enter, or by hitting escape. These interfaces are all modal, and when invoked they occupy large amounts of space on the screen.

For each of these mockups you can click through for a larger version.

Searching the Web (control+k, or alt-alt)

Of all of these mockups, I think keyboard-based Web search would be the most useful. This mockup also features some favicon upscaling code I wrote for another Mozilla Labs project.

I1Kuiwebsearch

In addition to Web search, the “Bookmarks and History” search will likely be more efficient than the current WIMPy ways of accessing bookmarks in Firefox:

The move back to language started with web search engines in general, with Google placing the capstone when its name became the house-hold verb for “typing to find what you want”. In fact, Googling is almost always faster then wading through my bookmark menu (which says there is something wrong with using menus as a mechanism for accessing bookmarks). (Command Line for the Common Man: The Command Line Comeback)

Switching Tabs (control+tab)

I1Kuitabswitching

Navigating Recent History

I1Kuirecenthistory

Tagging Pages

I1Kuitagging

Acting on Microformatted Content

I1Kuimicroformatdetection

Conclusion

-Just because the command line predated the graphical user interface doesn’t mean interfaces based on windows, icons, menus and pointers are always superior to interfaces based around using the keyboard for input.

-Designing interfaces based solely around the mouse and standard GUI widgets, and adding keyboard accelerators as an afterthought, does not always result in creating the most effective and streamlined user interfaces for advanced users.

-Interaction designers should consider designing keyboard-based graphical user interfaces, to simultaneously take advantage of both high bandwidth input, and high bandwidth output.

Technorati Tags:

57 comments

  1. This would be really wonderful !

  2. This looks really good, but fundamentally how different is it to the normal keyboard shortcuts? They’re still not discoverable.

  3. >fundamentally how different is it to the normal keyboard shortcuts?
    >They’re still not discoverable.

    In terms of initial discoverability, you’re right that these are essentially the same as normal keyboard shortcuts (especially since a lot of these interfaces might map to the existing shortcuts).

    However, after hitting the keyboard shortcut, subsequent keyboard actions do have visual affordances. For instance, in the Web search interface, the vertical order and arrows drawn on the interface may compel the user to try hitting the up and down arrow keys to select a different search engine. In Firefox’s current UI you can also set the search engine after hitting control+k, but there is no indication how to do this, or that this is even a possibility (control-up and control-down for anyone curious).

    In terms of overall discoverability, I’ve been thinking about displaying a list of possible keyboard-based interface shortcuts when the user hits the alt key. Hitting the corresponding letter would then launch the appropriate interface, and this would enable users to learn all of the direct alt-[letter] commands for the keyboard-based UIs.

  4. Really nice interface! Would love to have such keyboard shortcuts – seems like a lot easier than todays. This is a real killer feature just like tabs and it’s a cool way of showing microformats :)

    A thought about discoverability – in games I always look up keys in the settings-menus… Could that be useful to know here?

  5. Those mockups look great. I’m a Linux user envious of how good OS X always looks.

    Anyway, I’m glad to hear about elevating the importance of shortcuts. There are of course Vim like extensions, but leveraging the GUI along with the shortcuts gives more people what they want in an interface. I’ve just started porting Vim to run in Firefox, so I’d love to see some relevant code posted here doing what you are mocking up.

    Thanks.

  6. These look great, Alex. Very exciting!

    Incidentally, have you seen this extension?
    http://en.design-noir.de/mozilla/ctrl-tab/

    It’s a first step in the direction of the UI you suggest for tab-switching.

    This also reminds me of this article, out of the depths of time:
    http://acm.org/cacm/AUG96/antimac.htm

    especially the “See and Point” and “The Central Role of Language” sections.

  7. Sounds great, but _one_ thing disturbs me.

    This sort of feature (hell, that’s way more than a “feature”, let’s call it a concept) is great, but I don’t think it’s Mozilla’s task.

    Why ? Because let’s suppose this sort of feature pops out in Fx3. What will happen ? Emulation. A lot of coders will realize that’s great and implement on a per-case basis. Unstandardized, heterogeneous, and hopefully, after some years, some spineless consensus will appear.

    I think it should be the task of a group of UI / HID experts, then integrated in Mozilla through an API / standard.

    Reminds me of desktop notification. That’s a great concept, but each app make it its own way. Gnome apps are quite homogeneous, KDE apps too, the Mac has Growl, and Windows is far behind.

  8. > In terms of overall discoverability,
    > I’ve been thinking about displaying a list
    > of possible keyboard-based interface shortcuts
    > when the user hits the alt key.

    I’m sure you’ll be pleased to hear that this is something that the Newton developers did. If you press and hold the Apple key on a Newton or eMate keyboard for a few seconds, a list of keyboard shorcuts appears. By requiring the user to hold the key for a few seconds it doesn’t appear during normal use – but because all the shortcuts use the Apple key anyway, it’s not difficult to remember how to access this reminder, even if you can’t remember any other shortcuts.

  9. Really nice idea, interesting how looking back can give ideas for the future. I was just thinking about how this keyboard / command access could combine with gestural interfaces like MS Surface. The shortcuts providing a lightning quick filter on huge datasets, at which point you start dragging and rotating… I was just thinking how clunky it is to browse a large library of images in Lightroom – this sort of quick modal access would work really well.

  10. I thought of doing something like the Dao’s extension when I’ve seen the SeaMonkey’s tab preview.
    But I don’t have enough skills, indeed.

    I would love the idea of navigating trough the tabs using the titles or URL’s.

    > I’ve been thinking about
    > displaying a list of possible
    > keyboard-based interface shortcuts

    Well…
    If it’s an attribute, that’s fairly easy.
    I’ve a opaque letter, surrounded by {} with a tiny border (it becomes a little less opaque on mouseover) for access keys.
    It with CSS alone.

  11. Thanks for the tip about Enso Launcher! I think it was a great application.

    I think Firefox should stay out of this. It could potentionally be used as Opera does with Shift + Esc for access keys.

  12. Abdulkadir Topal

    Well, in fact Firefox has this already with quickfind, you invoke it with /. But of course as a quicksilver user I’d like to see more of it.

  13. I think this is an awesome idea, and i for one do think its firefox task to do stuff like this. Its part of the UI so why not ?

    Anyway the problem with this is that from what i know, keyboard shortcuts are mainly used by expert users.

    If you divide your uses into beginners / novice / expert, the novice part is by far the largest of the user base.

    This is mostly a focus problem then a UI problem…

    Personally i would love this feature.. !

  14. Glad someone in a position to make something thought of this. I noticed the trend 2 years ago.
    Good stuff!

  15. I would love to see this in FF. However, I’d like to point out that part of the wonder of Quicksilver is that there’s one interface for searching everything, and that interface is accessed through one easy-to-memorize keystroke. I envision something like that for FF, instead of having one interface for tabs, another for bookmarks, and a third for history. With one interface, you’d just start typing what you’re looking for, and it automagically knows what you’re talking about, no matter what type of data you had in mind.

  16. It looks really good, but fundamentally how diffrences are with the normal keyboard shortcuts?

  17. Regarding “searching the web” – this is one of the reasons that I just don’t use the new and cool search dialog. Using search keywords and typing alt-d “g expression” or alt-d “wp expression” is just so much easier than alt-k, use-the-mouse-to-switch-searching-engine, “expression”. While most of my searches are on google, I search the Wikipedia or addons.mozilla.org often enough that switching the search engine becomes an issue.

    If the functionality of the search box (like search suggestions) could be made to trigger in the main address bar after typing the proper keyword (or conversely, keywords could be used to switch the searching engine in the search box), now that, I think, would be a great usability enhancement.

  18. Brilliant idea Alex. Thanks for very interesting article. btw. I really enjoyed reading all of your articles. It’s interesting to read ideas, and observations from someone else’s point of view… makes you think more.

  19. hi alex, its a nice idea and a good article. nice work.

  20. Personally i would love this feature.. !

  21. Keyword commands seem to play nicely into this direction:

    http://portal.acm.org/citation.cfm?id=1166275

    ~L

  22. This looks excellent.

    For ideas on how to improve command discoverability, take a look at how Emacs does things. Every command is named and is invokable via Alt-x and then entering the name (offering tab-completion, of course), and you can look up commands by their name so that their bindings (if applicable) are shown. You can also get a listing at any time of all active bindings. Makes it very easy to explore.

    The eMate hold-the-modifier-key trick mentioned above is also a great idea.

  23. Anyone looked at cuiterm?

    Its a hybrid of GUI and cmdline

    unfortunately i think the author has moved to other projects, but i really LOVE the idea behind cuiterm. Maybe it all comes together… www+cuiterm :)

  24. Displaying the shortcuts by pressing the prefix key is fine, but it is not radically different from using the menu bar keys. Offering the option of configuring hot keys to the user would help to involve the user in the interface.
    One possible improvement to the GUI would be to have all menu entries, dialogs, combo boxes etc. assigned a number/letter/or? when a modifier key is pressed. Pressing that number jumps the focus to that element. It would improve navigation with the keyboard immensely and it would make scripting vastly simpler. It would also make user-built favorites or history easy to navigate.

  25. Thanks for very interesting article.

  26. Wow, this looks great!

    The changes for Tabs is similar to something I proposed [http://home.pacbell.net/eevans2/puzzler.htm] — see item #3 that suggested a dedicated Ctrl+Tab overlay supporting Find As You Type… but filtering as you type is much better.

    You mention that you “don’t have to hold down any keys for the window to stay up”. I think that would be a mistake since it makes it much slower to switch back to your last tab (you have to press Enter) and it’s inconsistent with Alt+Tab where you need to hold down Alt. One option is to support both modes and there are various ways this could be done (such as in the menus where you can click-and-drag or click twice), but I think Ctrl+Tab+release should stay as a quick way to switch to your last tab. (I mean stay the same as Alt+Tab not the current position-based Firefox behavior).

    Other suggestions:

    Put a label (“Filter:”) before the text entry box

    Include a large version of the Favicon next to each image

    Put the full title for the currently highlighted tab somewhere

    Consider including additional functionality in the overlay such as closing Tabs (similar to “Quick Tabs” in IE7). Once a user has pressed Ctrl+Tab, he or she has signaled interest in doing something related to tabs so a large portion of the screen can become dedicated to that task.

    You may also want to play around with Switcher [http://insentient.net/] since it’s an Alt+Tab replacement that supports filter as you type.

  27. One thing to consider is that even more people are getting onto the web everyday in the non-western world. Things that might seem easy to old time surfers might not be to someone who’s new to the web.

  28. You are absolutelly right man :)

  29. This looks really good, very good.

  30. People using only a keyboard often believe they are getting their tasks done much faster than people who have to move their hands back and forth to a mouse. They are sometimes right and sometimes not. Hand movement to and from a mouse is often concurrent with “think time,” thereby adding nothing to total task completion time. Conversely, the number of characters needed to remember and type a command and its parameters error-free, especially if they include file paths, often makes keyboard control much slower than a combination of keyboard and mouse in a well-designed interface.

    But the ability to operate software with nothing but a keyboard is very valuable for people (like programmers and game players) who repeat certain actions frequently and for people (including the blind) who have a reliable way to enter keystrokes but find it physically difficult or impossible to operate a pointing device.

    The more that interface designers pay attention to accessibility the more people who prefer keyboard control for other reasons will benefit.

  31. With apples launch of the voice recognition and dragon, i think we will eventually be using voice recognition software more then ever:-) and iphone in the mix

  32. I recently saw an article written by a professor, which confirm that what is written here. I agree with that.

  33. I’m sure you’ll be pleased to hear that this is something that the Newton developers did. If you press and hold the Apple key on a Newton or eMate keyboard for a few seconds, a list of keyboard shorcuts appears.

  34. I’ve tried the Control+Tab extension, and I have to say I found it annoying that I didn’t get instant feedback when switching tabs – the thumbnails were nice, but not as nice as seeing the actual tab which I’m about to use. In that sense, it actually felt less efficient to me. I think maybe if I really did have a lot of tabs, it would be more efficient, but I tend to use Ctrl-Tab when I know what I’m doing already, in which case the tab bar is good enough feedback for me.

  35. I agree with Ronan @7
    I really want a great gui/commandline interface. I also want the opportunity to choose the best one I can find or make.

    If mozilla makes a commitment to exposing functionality via API’s, then I can use the same command line for FF as I do for Thunderbird. And hey, maybe the Miro guys could expose functionality as well.

    That way, I don’t have to learn the tricks for each application, I just need to learn how to use Quicksilver or Enso or Launchy or Beagle…

  36. Yes. God, yes. Quicksilver, Enso, et. al. are by far my favorite way to interact with a computer. If every program had a system like that, then I would be in heaven.

    This should be an OS feature:
    Hit one keyboard shortcut for the OS native graphical keyboard (running programs, searching for files…) or hit a different graphical keyboard shortcut to access the functions of whatever program has focus (the same shortcut for any program).

  37. There is a open source tool called qliner hotkeys that does al this. You press Win+Z a keyboard pops up on screen (discoverability!). You can use this to do a quick lookup.

    The keyboard shows icons on configured keys. You can drag and drop stuff on the keyboard (shell items, links, etc.)

    It does smart starting, switching etc. You can do scripts.

  38. I must say that I’m extremely interested to give this a try. This looks very promising.

  39. I would like to introduce you to PodiPodi.com, inspired by Enso launcher but with a different approach. PodiPodi is served as a web service embeddable in a web site as a widget. The web designer can give his visitors an additional layer of interaction with a bunch of services (Google, Yahoo, Twitter, Flickr, YouTube and others) everyone can use it, no need for plugins or installations.
    Moreover webdesigner can expand their own PodiPodi instance by creating custom commands based on Google Gadgets.
    Embedding PodiPodi doesn’t force the web designer to modify his site’s layout.
    Maybe it won’t spread through the web but it’s an experimental project interaction that could drive to intersting results.

  40. Like me. Very useful advices. I think I will search more about this on the Net, Guys. Many thanks
    I like to cook very at home. Greetings from Poland. :)

  41. Heya Alex,

    I really enjoy your blog, and especially the ideas it is exploring, both in the articles you write, and the comments interaction afterwards.

    In extra defense of the terminal (as if it needed any :), and of keyboard accessible features, I wanted to point out that there are far more than 100M combos, even if you stick to only alphanumeric:

    “In just five alphanumeric characters, you can choose one out of 100,000,000 possible sequences.”

    Sure, if you count only lowercase letters, you have:

    (26 + 10) ^ 5 = 60,466,176

    But, if you include caps, you’ve got:

    (26 + 26 + 10) ^ 5 = 916,132,832

    And, smacking the shift key along with a letter probably won’t add to that 1 second time :)

  42. Good interface, nice keyboard shortcuts! Many Greets from Germany :-)

  43. Hello Alex,

    I really like your concepts. I agree with you on the quickness of typing a command versus point and clicking and just occurred to me that hardware improvements could also be a partner to your approach, when you have keyboards like the Optimus: http://www.artlebedev.com/everything/optimus/ the commands can be graphically displayed on the keys, instead of on the screen.

    Cheers,

  44. Let’s try Ubiquity. This is what you can do with it now:
    https://wiki.mozilla.org/Labs/Ubiquity/Ubiquity_0.1_User_Tutorial

    Cheers,
    -Bogumil

  45. I was just thinking how clunky it is to browse a large library of images in Lightroom – this sort of quick modal access would work really well.
    Anyway the problem with this is that from what i know, keyboard shortcuts are mainly used by expert users.

  46. This looks really good, very good.

  47. Touchscreens are what we need!!! That should combine the advantage of keyboard (being able to use our fingers) and GUI. Or maybe keyboards should be replaced by touchscreens. Those could display the keys on them during normal typing usage, or they could morph into a GUI hardware by displaying graphical symbols relevant to what you’re doing.

  48. Two comments:

    1) Want to use an AMAZING UI? Trade your soul to borrow your friend’s iPhone 3G for a day.

    2) Chrome has beaten Firefox to the punch of at least some of these issues. E.G.: standard address/URI/URL bar has been replaced with the super-powerfully box. Talk about, “Where do you want to go today?” It automatically searches your bookmarks, Google, and/or everywhere else to bring you instant, INTELLIGENT access to all of the above. Really.

    (“Define ‘intelligent’.”
    (“Correcting assessing the domain of recently visited websites and offering the domain before offering followed links of that domain that are cluttered with trash that directs you to your email inbox – which won’t work because you haven’t even signed into your email yet.”)

  49. So the take away lesson here is that everyone needs to learn how to use their chosen interface correctly.

    Truthfully, USERS DON’T WANT TO LEARN HOW TO USE A COMPUTER.

    Stupid but true.

    And your wasting time fighting that.

    This is why only “experts” know how to get around correctly – they are interested in learning the correct or best way of achieving their goal.

    As for touchscreens, if you have never searched for Microsoft Surface, better get started. This technology is quickly becoming less than five years away.

  50. Very nice site – congratulations

  51. 7 years ago I wished to use line commands for my computer because I liked the aesthetic of MS DOS with its black screens and white commands, but I don’t know anything about computers. Now, line commands and somehow black windows are being the future. That is very nice for me!

  52. Just downloaded the last version of Ubiquity and started to play around. I’m wondering how many shortcuts could be memorized by average user to make it efficient

  53. its a nice idea and a wonderful article. nice alex

  54. That’s perfect that we are able to take the mortgage loans moreover, this opens up new opportunities.

  55. Thanks for a great share of this blog.