Usability Testing

We’re performing another round of usability testing at Likewise. This is something that’s been critical to our success and, now that our products are getting more complex, it’s time to do it again.

I don’t remember exactly when usability testing came into vogue. I remember doing it way back in the early 1980s when I was working at Hewlett-Packard. This was in the early days of “WIMP” user-interfaces (Windows, Icon, Menu, Pointer). I remember asking users to edit some text in a word processor and they kept clicking on the “Edit” menu trying to find some editing commands other than “cut” and “paste”.

We had a nice lab at HP. This was in PSD, the “Personal Software”, division. We developed a suite of office software, HP Calc, HP Drawing Gallery, HP Access, etc., for the HP Touchscreen and, later, the Vectra personal computers.

The lab would simultaneously capture screen output and a video of the user’s facial expressions. These would be combined “in post” (postproduction) into a picture-in-picture composite videotape. You’d hear the user describing what he/she was asked to do (we asked them to vocalize their thoughts) and then you’d see how they’d try to achieve it using your UI. Their facial expressions were invaluable to understanding where they were getting confused.

At Microsoft (where I worked from 1987-1998), we did similar usability testing but we often farmed it out to other companies. The data capture was excellent but I thought that we missed out by not having developers present during the actual testing. I also think the company was less than diligent in learning from its testing.

I remember Microsoft hiring a couple of Stanford professors (Nass and Reeves) to design and evaluate our “social” user-interface: i.e. “Clippy” and kin. They’d designed the little popup characters that would “help” you in MS office. What I most remember about the professors was the startling data they’d collected when performing usability testing. The data said that the most popular “character” was “none.” Not Clippy. Not the dog (who came in a distant second). None.

Somehow, I expected this to pretty much kill the concept but, I guess, after you spend that much money on consultants you feel compelled to go on with it anyway. 

BTW, here’s a paper, by a Nass student, analyzing why people hate Clippy.

At Likewise, we don’t have money for a fancy lab but, what we lack in money, we make up for with enthusiasm. A video projector, computer and video camera are all it takes. What we do is to take the computer output and feed it to a video projector that shows the computer screen behind the test subject. We then aim the video camera at the user, simultaneously capturing the screen output behind the user. Total cost: $50 in gift certificates and $10 in pizza. Value: priceless.

I spent a little time experimenting with Camtasia as an alternative to our low-tech approach. Camtasia is an excellent “demo capture” program that can record screen activity directly to Flash video. It can also record microphone and video input, too. The video input is automatically composited as a picture-in-picture – exactly what I wanted!

Alas, the quality of the recorded video (the actual PIP camera-based video) is not as good as I’d like. The screen activity looks great but the camera input looks like 15 fps at best. We’re going to experiment with video settings and, maybe we can improve this. If we can get it to work, it’d be even easier than the projector setup that we’ve been using.

Update: we bought a cheapy Logitech web cam and are getting much better results than with our nice 3 CCD Sony video camera. It may be because we can better control the size and frame rate or, maybe it’s a codec thing. Regardless, with the new camera, the Camtasia approach works pretty well. As an “extra added bonus”, the cheapy web cam is able to perform face tracking. It has a little motor that pivots the camera around to face you. Effective, but creepy.

Don’t let the technology get in the way of the testing. At worst, you could have users bang on your software while you literally looked over their shoulders. It’d still be a valuable exercise. You’ll learn a tremendous amount by watching users struggle with things you thought were obvious. If you’re smart, you’ll also realize that, no matter what your excuses are, you’re wrong. The proof is in the video pudding. Fix it.

19 Responses to “Usability Testing”

  1. Świetny post. Bardzo ciekawie opisany artykuł. Jak możesz sprawdź stronę o samochodach u mnie! Do zobaczenia!

  2. Perfect piece of work you have done, this internet site is really cool with fantastic information.

  3. I am only writing to make you understand what a remarkable experience our princess undergone viewing your webblog. She picked up some things, which included what it is like to possess an ideal teaching spirit to make the mediocre ones without difficulty master specified tortuous things. You undoubtedly surpassed readers’ expected results. Thanks for rendering those useful, trustworthy, revealing as well as unique tips on your topic to Mary.

  4. I am also commenting to let you know of the helpful encounter my friend’s daughter found reading through your web page. She figured out numerous issues, not to mention what it’s like to have an amazing giving mood to have most people really easily know selected impossible subject areas. You really exceeded our own expectations. Many thanks for imparting the invaluable, safe, explanatory and even easy thoughts on the topic to Janet.

  5. Your mode of explaining everything in this paragraph is
    actually fastidious, all be able to easily be
    aware of it, Thanks a lot.

  6. Fantastic post but I was wanting to know if you could write a litte more on this subject?
    I’d be very thankful if you could elaborate a little bit further.

  7. says:

    Good day! Do you know if they make any plugins to safeguard
    against hackers? I’m kinda paranoid about losing everything I’ve worked hard on. Any suggestions?

  8. That is really attention-grabbing, You are an overly skilled blogger.
    I have joined your feed and look ahead to in the hunt for more of your great post.

    Additionally, I have shared your website in my social

  9. says:

    I don’t know if it’s just me or if perhaps everyone else experiencing issues with
    your blog. It looks like some of the written text in your posts are running off the screen. Can somebody else please comment and let me know if this is happening to them too?
    This could be a problem with my web browser because I’ve had
    this happen before. Thanks

  10. Wow, this post is good, my younger sister is analyzing these things, thus I am
    going to let know her.

Leave a Reply