We recently conducted a one week in-house usability test with Mozilla staff to gather feedback and problems with the current product so we could improve the experience.
We received great feedback and we wanted to share that information with you TowTruckers out there! Our users uncovered many problems (listed below) and we are working hard to fix these issues.
Here is a list (but not all) of the problems our users encountered during TowTruck Alpha user-testing.
- Users kept pressing / following the other person’s cursor to follow them around the page and to the next page, and wanted to chat with them that way. #500
- Users thought they needed to email to invite people to the session. #479
- Users kept trying to close the browser/tab to end the session. #442
- Users wanted to go the current position of the user their were following when they went to a new page. #489
- Users didn’t understand who the participants were in the dock (they thought that was their avatar). #475 and #312
- Users didn’t understand when a user was down the page (they thought the cursor was just sitting at the bottom of the chrome). #47
- Users accidentally ended a session, then created a new session, and then tried to send the old link. #456
- Users kept pressing the participant avatar in the dock because they thought that was their profile and wanted to change their name there. #453
We gathered ten participants (thank you user-testers!), who had various backgrounds, skill levels and prior knowledge of TowTruck. We then ran one-on-one sessions for approximately twenty minutes each. The last session we ran included two participants to test a real-time multi-user experience.
Some basic guidelines to follow as the moderator:
- Create a script with a natural flow, and come up with a scenario.
- Be neutral.
- Find mistakes in the flow.
- Do not ask leading questions.
- Give the participants clear tasks that are realistic to your product, e.g. “You’re having trouble finding a link on the page, so invite a friend to help you find it.”
- Try not to influence or direct.
- Record the sessions using video or a screen recorder.
- We gave each user two scenarios to walk through, that would emulate how they might use TowTruck in a real-world scenario. For example, we tried to model situations similar to coding collaboratively, working on a complicated math problem, or trying to decipher a big document like The Odyssey.
- Try not to the be a participant.
We went through a few scenario iterations, ranging from trying to use the Firefox Add-on, to cloning existing sites like kayak.com and adding TowTruck to it, to making fake-sites like these:
We ended up using the TowTruck Examples pages. This worked okay because there were fewer bugs, but there was some user confusion. For example, users had to imagine being in an actual scenario like “working on coding a website,” when they were on this sample Code Editor page which didn’t match interactions and context completely.
Here is the example for Scenario #1 instructions:
“I need help filling out this complicated form…” -> User experiences the Participant scenario.
- Join the session.
- Let me know who you are (change your name).
- Ask me what you want to do on the page (send a chat).
- End the session.
As the participant went through the tasks and scenario, they were asked to narrate their thought-process. We found specific pain-points and problems in the flow, from what they said aloud to just observing where they clicked on the screen. Some interactions were cumbersome and awkward, like “creating your profile name” and some UI elements were confusing like the “hangup button.”
During one of the sessions, we “declined to join a session.” This was never tested before and we found all sorts of bugs and problems with that flow.
Normally, it’s best to cluster the feedback with post-it notes and add them to something like Evernote, but wanting to move fast, we threw the notes in Evernote, then went back and watched through the videos to double-check them. The notes were then converted into Github Issues with the Issue tag “user-feedback-Alpha” here.
It’s always great to have some fresh eyes look at the product, use it and break it. Often, when heads-down creating the product, the product-creators get used to certain flows, UI elements, and interactions that they don’t feel strange/awkward anymore, and subconscious work-arounds are created to complete those tasks.
Other user-testing insights:
- We started to get the same issues and problems popping up when we hit the 6-8 participant threshold, which is a known issue.
- TowTruck was handy for usability testing because it shows mouse-clicks.
- We ended up using the TowTruck Examples page, which was okay this time, but it would be better to place TowTruck on sites where users would actually find TowTruck useful.
- Team TowTruck