Usability Testing, Revisited

A couple of weeks ago, I wrote about usability testing. Mostly, I talked about testing methodology and the value that it brings. We’ve now finished 8 sessions and I thought it would be good to revisit the subject.

Once again, I’m amazed by how much value usability testing can provide. We’ve been testing our Likewise Open evaluation and download process with the goal of increasing the number of people who successfully install the software. This process begins with a user arriving at our web site and ends with that user performing a successful “join” operation to connect his/her non-Windows computer to Microsoft Active Directory. When we first decided to test the process, I thought we’d not learn much. What could be simpler than clicking on a download link and running an installer program? For the 1,498,753rd time in my life, I was wrong.

The first thing that we learned is that our home page is not our Home page.

Although we’ve tried several web analytics packages, we’ve lately becomed enamored with Google Analytics. The free version is relatively capable and sufficient for our current needs.  A few weeks of analysis with the tool told us that more customers are coming to our Likewise Open community page than to our corporate home page. Looking at the analytics report, it was obvious why: our partners are driving traffic to our site and their linking to the Likewise Open page instead of the corporate home page.

This makes sense. When our Linux partners want to reference Likewise, they want to get their customers as close to their final destination as possible. They don’t want to link to a high-level page with a lot of sales-oriented material. By linking to our Likewise Open community page, they are taking users to a page that’s very relevant and only a couple clicks away from a download.

When we realized that our Likewise Open page was our effective home page, we realized we needed to improve it. We knew that it would be a bad idea to make it too sales-oriented but we also knew that it had several shortcomings. This was borne out in usability testing and quickly corrected.

The second thing we learned is that clicking on a link is non-trivial.

Our download page has a big table with many different rows for different operating systems (Linux, Solaris, etc.), different CPUs (i386, SPARC, Itanium, etc.), different CPU modes (32/64 bit) and different packaging forms (RPM, DEB, etc.). The user has to find the right row and click on a download link. Simple no? No.

If you don’t set your mime-types properly on your web page links, Firefox can make a mess of things. We had many complaints of users who would get a screenful of binary stuff instead of a downloaded file.

The next thing we learned is that it’s possible to be too smart.

Linux and UNIX folk are used to painful install processes. They’ll download packages and then have to use some type of package manager (rpm, dpkg, etc.) to install it. We decided to make life easier for customers by giving them a nice, executable, installer program. In the case of Linux, Mac and other operating systems that are likely to have a GUI present, we make use of a Bitrockbased setup program. Download the software, run it. Simple no? No.

When Firefox downloads a program to Linux, it doesn’t retain its executable file mode. Before you can run it, you have to chmod +x it. If users didn’t read our 100 page Installation and Administration Guide they might not realize this. In fact, as usability testing pointed out, they might try to do other weird stuff.

Our setup programs are typically called something like LikewiseOpen- Long, yes, but it tells you everything you need to know: product name, version, operating system, architecture and packaging format. Note that we include “rpm” (or “deb” or other) in the name. Some Linux folk would fail to realize that the installer was an executable, would see the “rpm” in the name and think, “maybe I’ve got to install this thing with the rpm program.” Wrong.

The last thing that we learned is that nobody reads anything. No documentation, for sure. They don’t spend much time reading screen output, either.

After users install the software, they need to run our domain-join utility afterwards. We tell them this at the very end of the installer program. Alas, as usability testing showed us, many users decide to just ignore that information and hit Enter repeatedly without reading anything. Right after they dismiss the last dialog they realize that they just missed something important.

We’ve made numerous changes to the Likewise Open pages as a result of our testing. Much of it has been simple to accomplish: more prominent links; short, task-specific document; a short video; corrected mime-types. We’ll do a new round of usability testing now to verify the results of our changes. I’m confident that we’ll see improvement but I’m much less confident now that we won’t find a different set of problems to address. The main lesson of usability testing is that your software UI is never as good as you think it is.

13 Responses to “Usability Testing, Revisited”

  1. Like!! Thank you for publishing this awesome article.

  2. I like the valuable information you provide in your articles.

  3. Good one! Interesting article over here. It’s pretty worth enough for me.

  4. SMS says:

    bookmarked!!, I like your blog!

  5. lost vw key says:

    I like the efforts you have put in this, thank you for all the great content.

  6. cbd oil says:

    I intended to compose you a bit of note in order to give many thanks again considering the beautiful concepts you’ve contributed above. It’s extremely generous of people like you to give openly just what many individuals might have marketed as an e book to generate some dough for themselves, most notably given that you could possibly have done it if you ever decided. These points as well served to be a fantastic way to fully grasp other people have similar fervor much like my own to know the truth lots more pertaining to this issue. I am certain there are millions of more fun opportunities in the future for individuals who looked at your blog post.

  7. I precisely desired to say thanks again. I’m not certain the things I could possibly have achieved in the absence of the aspects contributed by you about my area. Completely was a challenging case for me, but being able to view a expert approach you dealt with it forced me to cry for contentment. Extremely grateful for the advice and believe you are aware of a powerful job that you’re accomplishing educating many others all through a blog. I’m certain you’ve never come across all of us.

  8. says:

    Hmm it looks like your website ate my first
    comment (it was super long) so I guess I’ll just sum it up what I
    submitted and say, I’m thoroughly enjoying your blog.
    I too am an aspiring blog writer but I’m still new to everything.
    Do you have any tips for rookie blog writers? I’d definitely appreciate it.

  9. I like the helpful information you supply in your articles.
    I’ll bookmark your weblog and test once more here regularly.
    I’m reasonably sure I’ll be informed many new stuff proper
    here! Best of luck for the next!

  10. says:

    I’m really enjoying the design and layout of your site.
    It’s a very easy on the eyes which makes it much more enjoyable for me
    to come here and visit more often. Did you hire out a designer
    to create your theme? Excellent work!

  11. Very well written post. It will be helpful to everyone who utilizes it, including yours truly :). Keep up the good work – for sure i will check out more posts.

  12. zortilo nrel says:

    I’ve recently started a website, the information you offer on this web site has helped me greatly. Thank you for all of your time & work.

  13. says:

    Does your website have a contact page? I’m having
    a tough time locating it but, I’d like to shoot you an e-mail.
    I’ve got some creative ideas for your blog you might be interested
    in hearing. Either way, great website and I
    look forward to seeing it develop over time.

Leave a Reply