And Then…

I had the opportunity to spend a few hours at Oscon yesterday in Portland, Oregon. Oscon is the Open Source Conference held by O’Reilly. I was pleasantly surprised by the size of the conference, the number of exhibitors and the presence of several large companies. Open source software has definitely become mainstream and accepted by industry.

At Likewise, we consider ourselves an open source company. Likewise Open has been very successful and has opened many doors for us (no pun intended). It’s helped us tremendously, even when we end up selling our Enterprise version instead. Nevertheless, I have some observations about open source, not all of them positive.

There are several definite advantages to using open source. It enables you to build a solution without having to reengineer every component. We make use of both MIT Kerberos and OpenLDAP in our products. If we had needed to rewrite these components, it would have taken us much longer to get to market. We’ve also made use of Samba components. Samba has been around a long time, has had “a lot of eyes” on it and has figured out the subleties of talking to Microsoft systems. Again, using open source saved us a lot of time.

There are some disadvantages to open source, too. It can be difficult to get the “owners” of an open source project to do what you think is the right thing. Although open source is “open”, certain projects are led by designated groups of people. Different projects have different guidelines around software submission and how they go about accepting external contributions. Very often, your contributions have to be vetted before they’re accepted in the main code. If your code is not accepted, your only option is to distribute your own modified version of the open source project (your branch). Branching is not a good thing.

Sometimes, code changes are rejected due to style considerations or differences in design approaches. These are objections that can be dealt with relatively easily. More difficult are rejections due to “dogma”. Some open source projects, for example, are irrationally opposed to anything that they perceive as helping Microsoft. Even our intent is to make non-Windows systems work better they still oppose our goal of making these systems work better with Microsoft Active Directory. This, of course, doesn’t apply to the Samba project (who had the goal before we did) but applies to other open source projects/companies/teams with which we’ve had to deal.

There is little we can do in these cases other than to develop our own alternatives.

Another issue which we’ve encountered with some open source software is a certain lack of industrial rigor. I’ve worked a lot with both commercial software developers (I spent 11 years at Microsoft) and with academic programmers (4 years at Microsoft Research). Sometimes, open source software sometimes resembles the latter more than the former.

What do I mean by “academic” programmers? Say that you’re in school, you take a programming course and you’re asked to write a program that converts degrees from Celsius to Fahrenheit. You write something like: 

void main(int c, char **argv)
{
    int degrees = atoi(argv[1]);
    printf("%d Celsius is %g Farhenheit\n", degrees, (degrees * 9.0)/5.0 + 32);
}

Your professor would probably give you a passing grade for this. It works. In industry, however, your boss would likely complain about several things:

• Crappy user interface. How is the customer supposed to know that the input should appear on the command line?
• Poor error handling. What happens if you don’t supply a command-line argument? What if you specify a non numeric value?
• Bad spelling
• Lack of localization support
• Lack of comments in code

Open source software is not always industrial quality code. We have found many cases of memory corruption and leakage even in mature open source projects. We have also found and fixed many, many, bugs.

Note that the title of this post does not suggest that proprietary software is immune from similar flaws. Many proprietary software companies (including my ex-employers) are guilty of releasing software that is not ready for prime time. “Good Software” can be either open source or proprietary. Similary, “Bad Software” does not care about its licensing model.

What I will suggest, however, is that companies that have to support their products, keep customers happy and, ultimately, make money are much more motivated to develop Good Software than organizations which develop software but don’t actually have to deal with the consequences of poor code. There is no stronger motivator to write Good Software than an irate customer.

We are heavy users of virtualization at Likewise Software. Since we develop software for over 100 different platforms (multiple flavors of UNIX, Linux and Mac OS X), we have to be able to boot up a Red Hat 2.1 machine one minute and a Open Solaris machine the next. Developers and testers, both, need access to a wide variety of machines on a regular basis. Without virtualization, it would either be very expensive (we’d need hundreds of machines) or very slow (we’d have to re-image machines all the time) in order to do our work.

We also use virtualization outside of development/test. Over time, we’ve tended to collect an assortment of servers running project management tools, bug databases, internal wikis, HR, financial and other applications. A few months ago, our IT folk examined all the servers in our inventory and migrated many of them to virtual machines.

Unquestionably, virtualization can bring about good things — reduced administrative costs, increased flexibility, reduced energy use, etc. Virtualization doesn’t always make sense, however.

Occasionally, I have a conversation with someone who’s basically saying something like “Virtualization is terrible! I moved my database server and my risk management grid onto VMs and now they run at half the speed they used to!”. Yes, I do want to whack them upside the head when they say this.

Obviously, if you have a CPU-intensive, heavily threaded, application running on a physical server it’s going to slow down if you put it on a virtualized server along with other CPU-intensive applications. If you wouldn’t run these two apps on the same physical server, certainly, don’t run them on two VMs on a single physical server. VM hypervisors can run multiple virtualized machines effectively and with little degradation in performance, but only to the extent that the virtualized systems are amenable to this. If the VMs are running applications that are not heavily threaded and do not heavily tax their CPU and I/O systems then the VM hypervisor can exploit multiple cores and spare CPU cycles to provide acceptable performance.

There are some “textbook” examples of applications/systems that are ideal for virtualization. Web farms, for example, can deploy web sites in their own VMs and give you complete control of a virtualized server. You can muck with system configuration to your heart’s content without worrying about other web sites that might be deployed on the same physical server. Web farms can also quickly duplicate VMs allowing them to provide additional load-balanced capacity on an on-demand basis.

Beyond the textbook examples, here are some others to consider.

Infrequently run applications are great candidates for virtualization. Consider financial apps that might only be run at quarter- or year-end.  Rather than dedicating a machine to these applications that sits idle 95% of the time, these applications can be deployed on virtual systems that are suspended until needed. This approach is ideal for sensitive applications such as financial and reporting systems. It is best to not run these applications on shared hardware. If there are other applications on the same computer this increases the likelihood of intential or unintential access to secure data. With virtualization, physical systems don’t have to be “wasted” on infrequently used sensitive applications. Note, too, that by suspending sensitive VMs while they’re not in use that you’re reducing the attack surface for hackers.

Another great use of virtualization is for old, legacy, systems. If you’re running old versions of Windows NT or SUSE Linux or Solaris x86 and don’t want to update them (why fix something that’s not broken?) why not move these systems to VMs? In all likelihood, these systems are running on flaky outdated (perhaps unsupported) hardware. It’s possible that they’ll run faster on VMs than on old metal.

Demo systems are ideal candidates for virtualization. The systems receive a lot of “wear and tear” – they’re frequently polluted with sample data and often left in weird states. Moving these to VMs allow you to use VM snapshots  to quickly restore them to a recognizable state.

Finally, one of my favorite uses for VMs is as security honeypots. Create a VM (especially a Windows VM) and give it a suggestive name, perhaps, payroll or HR. Create some directories and files in it, again, with suggestive names. Now, turn on all the auditing features available in the OS. Protect this system as you would any other secure server in your network (but don’t use the same admnistrative passwords!). If possible, isolate this VM from your other systems. Put it on its own subnet and disallow routing to other systems, for example. If you have an intrusion detection system, make sure it monitors this VM. There should be no access to this computer (other than by you, to assure its health). If your IDS or audit logs signal that someone is trying to access the system, you know you’re under attack.

Virtualization has been around for 30+ years. I used VM/370 in college in 1977. It offers many benefits that, thanks to VMWare, Xen and others, are now available to any computer user. At the end of the day, however, virtualization is simply multitasking with really, really, good application isolation. Rather than multitasking applications that call a single operating system instance, hypervisors multitask entire operating system instances. The rest of the gory details (how they virtualize hardware, where drivers live, etc.) are just that: details.

A couple of weeks ago, I wrote about usability testing. Mostly, I talked about testing methodology and the value that it brings. We’ve now finished 8 sessions and I thought it would be good to revisit the subject.

Once again, I’m amazed by how much value usability testing can provide. We’ve been testing our Likewise Open evaluation and download process with the goal of increasing the number of people who successfully install the software. This process begins with a user arriving at our web site and ends with that user performing a successful “join” operation to connect his/her non-Windows computer to Microsoft Active Directory. When we first decided to test the process, I thought we’d not learn much. What could be simpler than clicking on a download link and running an installer program? For the 1,498,753rd time in my life, I was wrong.

The first thing that we learned is that our home page is not our Home page.

Although we’ve tried several web analytics packages, we’ve lately becomed enamored with Google Analytics. The free version is relatively capable and sufficient for our current needs.  A few weeks of analysis with the tool told us that more customers are coming to our Likewise Open community page than to our corporate home page. Looking at the analytics report, it was obvious why: our partners are driving traffic to our site and their linking to the Likewise Open page instead of the corporate home page.

This makes sense. When our Linux partners want to reference Likewise, they want to get their customers as close to their final destination as possible. They don’t want to link to a high-level page with a lot of sales-oriented material. By linking to our Likewise Open community page, they are taking users to a page that’s very relevant and only a couple clicks away from a download.

When we realized that our Likewise Open page was our effective home page, we realized we needed to improve it. We knew that it would be a bad idea to make it too sales-oriented but we also knew that it had several shortcomings. This was borne out in usability testing and quickly corrected.

The second thing we learned is that clicking on a link is non-trivial.

Our download page has a big table with many different rows for different operating systems (Linux, Solaris, etc.), different CPUs (i386, SPARC, Itanium, etc.), different CPU modes (32/64 bit) and different packaging forms (RPM, DEB, etc.). The user has to find the right row and click on a download link. Simple no? No.

If you don’t set your mime-types properly on your web page links, Firefox can make a mess of things. We had many complaints of users who would get a screenful of binary stuff instead of a downloaded file.

The next thing we learned is that it’s possible to be too smart.

Linux and UNIX folk are used to painful install processes. They’ll download packages and then have to use some type of package manager (rpm, dpkg, etc.) to install it. We decided to make life easier for customers by giving them a nice, executable, installer program. In the case of Linux, Mac and other operating systems that are likely to have a GUI present, we make use of a Bitrock–based setup program. Download the software, run it. Simple no? No.

When Firefox downloads a program to Linux, it doesn’t retain its executable file mode. Before you can run it, you have to chmod +x it. If users didn’t read our 100 page Installation and Administration Guide they might not realize this. In fact, as usability testing pointed out, they might try to do other weird stuff.

Our setup programs are typically called something like LikewiseOpen-4.1.0.2921-linux-i386-rpm-installer. Long, yes, but it tells you everything you need to know: product name, version, operating system, architecture and packaging format. Note that we include “rpm” (or “deb” or other) in the name. Some Linux folk would fail to realize that the installer was an executable, would see the “rpm” in the name and think, “maybe I’ve got to install this thing with the rpm program.” Wrong.

The last thing that we learned is that nobody reads anything. No documentation, for sure. They don’t spend much time reading screen output, either.

After users install the software, they need to run our domain-join utility afterwards. We tell them this at the very end of the installer program. Alas, as usability testing showed us, many users decide to just ignore that information and hit Enter repeatedly without reading anything. Right after they dismiss the last dialog they realize that they just missed something important.

We’ve made numerous changes to the Likewise Open pages as a result of our testing. Much of it has been simple to accomplish: more prominent links; short, task-specific document; a short video; corrected mime-types. We’ll do a new round of usability testing now to verify the results of our changes. I’m confident that we’ll see improvement but I’m much less confident now that we won’t find a different set of problems to address. The main lesson of usability testing is that your software UI is never as good as you think it is.