We’re performing another round of usability testing at Likewise. This is something that’s been critical to our success and, now that our products are getting more complex, it’s time to do it again.

I don’t remember exactly when usability testing came into vogue. I remember doing it way back in the early 1980s when I was working at Hewlett-Packard. This was in the early days of “WIMP” user-interfaces (Windows, Icon, Menu, Pointer). I remember asking users to edit some text in a word processor and they kept clicking on the “Edit” menu trying to find some editing commands other than “cut” and “paste”.

We had a nice lab at HP. This was in PSD, the “Personal Software”, division. We developed a suite of office software, HP Calc, HP Drawing Gallery, HP Access, etc., for the HP Touchscreen and, later, the Vectra personal computers.

The lab would simultaneously capture screen output and a video of the user’s facial expressions. These would be combined “in post” (postproduction) into a picture-in-picture composite videotape. You’d hear the user describing what he/she was asked to do (we asked them to vocalize their thoughts) and then you’d see how they’d try to achieve it using your UI. Their facial expressions were invaluable to understanding where they were getting confused.

At Microsoft (where I worked from 1987-1998), we did similar usability testing but we often farmed it out to other companies. The data capture was excellent but I thought that we missed out by not having developers present during the actual testing. I also think the company was less than diligent in learning from its testing.

I remember Microsoft hiring a couple of Stanford professors (Nass and Reeves) to design and evaluate our “social” user-interface: i.e. “Clippy” and kin. They’d designed the little popup characters that would “help” you in MS office. What I most remember about the professors was the startling data they’d collected when performing usability testing. The data said that the most popular “character” was “none.” Not Clippy. Not the dog (who came in a distant second). None.

Somehow, I expected this to pretty much kill the concept but, I guess, after you spend that much money on consultants you feel compelled to go on with it anyway. 

BTW, here’s a paper, by a Nass student, analyzing why people hate Clippy.

At Likewise, we don’t have money for a fancy lab but, what we lack in money, we make up for with enthusiasm. A video projector, computer and video camera are all it takes. What we do is to take the computer output and feed it to a video projector that shows the computer screen behind the test subject. We then aim the video camera at the user, simultaneously capturing the screen output behind the user. Total cost: $50 in gift certificates and $10 in pizza. Value: priceless.

I spent a little time experimenting with Camtasia as an alternative to our low-tech approach. Camtasia is an excellent “demo capture” program that can record screen activity directly to Flash video. It can also record microphone and video input, too. The video input is automatically composited as a picture-in-picture – exactly what I wanted!

Alas, the quality of the recorded video (the actual PIP camera-based video) is not as good as I’d like. The screen activity looks great but the camera input looks like 15 fps at best. We’re going to experiment with video settings and, maybe we can improve this. If we can get it to work, it’d be even easier than the projector setup that we’ve been using.

Update: we bought a cheapy Logitech web cam and are getting much better results than with our nice 3 CCD Sony video camera. It may be because we can better control the size and frame rate or, maybe it’s a codec thing. Regardless, with the new camera, the Camtasia approach works pretty well. As an “extra added bonus”, the cheapy web cam is able to perform face tracking. It has a little motor that pivots the camera around to face you. Effective, but creepy.

Don’t let the technology get in the way of the testing. At worst, you could have users bang on your software while you literally looked over their shoulders. It’d still be a valuable exercise. You’ll learn a tremendous amount by watching users struggle with things you thought were obvious. If you’re smart, you’ll also realize that, no matter what your excuses are, you’re wrong. The proof is in the video pudding. Fix it.