A couple of weeks ago, I wrote about usability testing. Mostly, I talked about testing methodology and the value that it brings. We’ve now finished 8 sessions and I thought it would be good to revisit the subject.
Once again, I’m amazed by how much value usability testing can provide. We’ve been testing our Likewise Open evaluation and download process with the goal of increasing the number of people who successfully install the software. This process begins with a user arriving at our web site and ends with that user performing a successful “join” operation to connect his/her non-Windows computer to Microsoft Active Directory. When we first decided to test the process, I thought we’d not learn much. What could be simpler than clicking on a download link and running an installer program? For the 1,498,753rd time in my life, I was wrong.
The first thing that we learned is that our home page is not our Home page.
Although we’ve tried several web analytics packages, we’ve lately becomed enamored with Google Analytics. The free version is relatively capable and sufficient for our current needs. A few weeks of analysis with the tool told us that more customers are coming to our Likewise Open community page than to our corporate home page. Looking at the analytics report, it was obvious why: our partners are driving traffic to our site and their linking to the Likewise Open page instead of the corporate home page.
This makes sense. When our Linux partners want to reference Likewise, they want to get their customers as close to their final destination as possible. They don’t want to link to a high-level page with a lot of sales-oriented material. By linking to our Likewise Open community page, they are taking users to a page that’s very relevant and only a couple clicks away from a download.
When we realized that our Likewise Open page was our effective home page, we realized we needed to improve it. We knew that it would be a bad idea to make it too sales-oriented but we also knew that it had several shortcomings. This was borne out in usability testing and quickly corrected.
The second thing we learned is that clicking on a link is non-trivial.
Our download page has a big table with many different rows for different operating systems (Linux, Solaris, etc.), different CPUs (i386, SPARC, Itanium, etc.), different CPU modes (32/64 bit) and different packaging forms (RPM, DEB, etc.). The user has to find the right row and click on a download link. Simple no? No.
If you don’t set your mime-types properly on your web page links, Firefox can make a mess of things. We had many complaints of users who would get a screenful of binary stuff instead of a downloaded file.
The next thing we learned is that it’s possible to be too smart.
Linux and UNIX folk are used to painful install processes. They’ll download packages and then have to use some type of package manager (rpm, dpkg, etc.) to install it. We decided to make life easier for customers by giving them a nice, executable, installer program. In the case of Linux, Mac and other operating systems that are likely to have a GUI present, we make use of a Bitrock–based setup program. Download the software, run it. Simple no? No.
When Firefox downloads a program to Linux, it doesn’t retain its executable file mode. Before you can run it, you have to chmod +x it. If users didn’t read our 100 page Installation and Administration Guide they might not realize this. In fact, as usability testing pointed out, they might try to do other weird stuff.
Our setup programs are typically called something like LikewiseOpen-4.1.0.2921-linux-i386-rpm-installer. Long, yes, but it tells you everything you need to know: product name, version, operating system, architecture and packaging format. Note that we include “rpm” (or “deb” or other) in the name. Some Linux folk would fail to realize that the installer was an executable, would see the “rpm” in the name and think, “maybe I’ve got to install this thing with the rpm program.” Wrong.
The last thing that we learned is that nobody reads anything. No documentation, for sure. They don’t spend much time reading screen output, either.
After users install the software, they need to run our domain-join utility afterwards. We tell them this at the very end of the installer program. Alas, as usability testing showed us, many users decide to just ignore that information and hit Enter repeatedly without reading anything. Right after they dismiss the last dialog they realize that they just missed something important.
We’ve made numerous changes to the Likewise Open pages as a result of our testing. Much of it has been simple to accomplish: more prominent links; short, task-specific document; a short video; corrected mime-types. We’ll do a new round of usability testing now to verify the results of our changes. I’m confident that we’ll see improvement but I’m much less confident now that we won’t find a different set of problems to address. The main lesson of usability testing is that your software UI is never as good as you think it is.