And Then…

In between eating hot dogs and blowing up fireworks this weekend, I worked on a couple of home IT projects that I’d been planning for a while. My goals were straightforward. First, I wanted to implement a more robust backup solution. Second, I wanted to get rid of a Fedora Core 5 server and replace it with something newer. The two projects were related since the FC5 server was being used solely as a Samba file server to host my backup drives. Here’s what I ended up doing to accomplish both tasks.

I didn’t like the dedicated Fedora server for two reasons. First, I was stuck using a computer for a very narrow purpose. I have only two computers in my “server room” (my den) and my other one is my AD domain controller. I’ve been installing lots of Windows application software on the AD machine because I can’t run it on Linux. Installing random software on a DC is not a good idea. The second reason I wanted to get rid of the Fedora machine is because I wanted to run a more current distro. I worried about replacing FC5 with Ubuntu, however, because FC5 uses a funky logical volume manager. If the change failed, I might have to scramble to recover my data.

To get rid of the FC5 file server, I spent $150 on a Linksys NAS 200 device and $200 on two 500Gb SATA hard disks. The Linksys device is, essentially, a cheapy Linux box with an ethernet port and two SATA drive bays. It can be configured to use the drives separately or in a RAID 0 (striping) or RAID 1 (mirroring) array. I chose the latter configuration giving me 500Gb of storage but with the security of knowing that I can lose a drive and still have my data.

The Linksys NAS 200 was pretty easy to install. I made one mistake which was to start using it (copying over 100Gb to it) before realizing that it was running very slowly. A look at the Linksys web site showed that there was a firmware update that allowed the use of a non-journaled file system. Without journaling, the Linksys device is much faster but will have to perform a “scandisk” (fsck) if detects any disk errors. Installing the firmware upgrade and switching to the non-journaled file system required reformatting the disks and re-copying the 100Gb again.

WIth the NAS in place, I was able to go to my FC5 computer and copy over all the old backups. I then changed the key computers in the house to use the NAS instead of the FC5 machine for backups. Along the way, I also stopped using Windows backup software and started using NTI Shadow (dumb, cheap) instead.

Now that my FC5 computer was out of a job, I could repurpose it. I increased its RAM to 2Gb, deleted its Linux file system partitions and installed Windows XP on it. Deleting the partitions was necessary as, with them,  Windows XP would get confused during installation.

The first thing I did after installing XP (well, the second, after waiting for SP2 and a million other updates to install), was to install VMWare Workstation. VMWare Server is free, but the Workstation version allows for multiple “snapshots” which I find very useful.

With VMWare installed, the first VM I created was an Ubuntu 8.04 Linux VM.

What’s the point of replacing a Linux machine with a Windows machine running Linux in a VM? Two things: first, I can run Windows software in the host operating system. Actually, I will probably create a Windows XP VM and run the Windows software in the VM instead of on the host OS. Second, if I get tired of one Linux distribution, I can always create another VM with a different one.

With VMWare, I can keep my host OS in pristine condition. I won’t install any application software there. If any problem occurs in a guest VM, I can always use the VMWare snapshot features to “undo” them. Worst case, I can blow away a VM and recreate it. What about data? Here’s the key: don’t keep your important data on virtual disks. Use virtual machines, but keep your data on real drives or on a NAS device. Windows file shares or the Linux mount.cifs command can help with this (if you keep all your data on WIndows file servers; if you want, you can use NFS and store data on UNIX file servers, instead). Use virtual disks only to store operating system files.

This is exactly the architecture used in large virtualized Enterprise IT departments. Application data is kept on attached storage accessed by one or more virtual machines. Deploying additional virtual server instances is easy because the data is centrally located. The same concept can be used at home, on a smaller scale.

Everything is up and running now. I’m happy running Ubuntu instead of FC5 and I’m happy knowing my data backups are mirrored. I’ll be tracking the performance of the NAS device over the next few weeks and months. Consumer NAS devices are a tricky tradeoff of simplicity vs. functionality and performance. Someday, I want to experiment with removing a drive from the array and validating that the RAID rebuild occurs properly. For now, I’m just hoping the NAS is doing the right thing.

This post might more accurately refer to Totalitarianism instead of Socialism but what it lacks in precision, it makes up for with alliteration!

July 4th! Independence Day. In Seattle, we also refer to it as “the day before summer begins.” It always rains here on July 4th. It’s a tradition.

Beyond its meteorological implications, July 4th commemorates the signing of the Declaration of Independence. 232 years ago, representatives from the original 13 colonies formalized their desire to secede from the British Empire. What beef did they have with King George? Why all the fuss?

True, we know they were upset about “taxation without representation” and about the Stamp Act and import tariffs. Did you know, however, that Britain had ceded on all these points? Did you know that the 13 colonies had the highest standard of living in the world at the time and a very low tax rate?

Beyond any concrete economic or political issues, the colonies wanted to secede from Britain because they resented being told what to do by a remote sovereign that treated them as second-class citizens. If you read the Declaration of Independence and, even more so, the Constitution, you will easily detect a fundamental distrust of centralized Federal government. The forefathers went out of their way to delegate the least number of powers to the Federal government – the rest were reserved for the states. Arguably, the 2nd Amendment is all about the rights of the States to maintain militias so that they could fight the Federal government. Remember, the militias were formed to fight the British. The last thing the colonies wanted was to be powerless against another powerful central government that might turn out to be just as objectionable as the first.

This fundamental question of State vs. Federal rights is one that remains with us today. Battles over abortion, Medicaid payments, “No Child Left Behind” and other issues focus on what are state responsibilities and what are Federal responsibilities. The States rights proponents argue that local government is more efficient and more responsive to local needs than a centralized Federal government.

Centralized vs. distributed control is also a frequent topic of discussion in Socialist vs. Capitalist systems. In Socialist systems, the State owns all the capital and decides how to allocate it. In Capitalist systems, individuals own capital and they decide how to invest it. Again, the question is one of tight, centralized, control vs. a loose distributed system.

This same tension exists in large software architecture. To what degree should software be tightly controlled by some central program vs. loosely controlled and autonomous? Occasionally, I will review a design for a complex system and I will compain to the architect that the design is “too Communist.” What I mean by this is that it too heavily relies on central planning and control.

As with governments and economies, there is a strong argument that software architecture should avoid excessive dependence on centralized control. Centralized control can lead to very brittle software that breaks when an alternative design would simply bend.  Centralization frequently translates to designs with single points of failure. A server goes offline or a process fails and the whole system collapses.

As one who dislikes both centralized designs and socialist governments, I like to see software architectures with:

• Redundant, clustered, storage (e.g. from Isilon)
• Compute and database clusters based on active/active designs to minimize failover time
• Pull-based models where automonous components ask for what they need instead of push-based models where an executive tells the others what to do
• Self-organizing systems rather than ones driven by rigid configuration

It is my premise that such systems are no harder to develop than centralized ones but they do require more creative architects to design them. Because we tend to design systems in a hierarchical fashion, it’s easiest to develop control systems that also flow from top-to-bottom. Developing loosely coupled systems takes a different mind-set. Once you’ve identified the components in your design, you need to treat each as an independent entity and you need to think about how they get the information they need and what they do with it when they’ve finished processing it. You also need to consider what each entity should do when it detects and error condition. If you design your components to be independent and robust, your design might be better able to heal itself if a component has an intermittent failure.

It is true that, in the physical world, there are economies of scale but it is also true that past a certain point, inefficiency and bureaucracy increase with organizational size. In the software world, there are few benefits to centralization and size and many clear deficiencies. As the computer and IT industry moves increasingly towards outsourcing and SaaS, we need to keep in mind to what degree delegating responsibilities to service providers is a move towards centralization and all of its flaws.

We’ve been doing a lot of interviewing lately, looking for developers, QA folk and deployment engineers. We’ve looked at hundreds (if not thousands) of resumes, performed numerous phone and live interviews but made only a handful of offers. It’s been difficult to find people with the skills that we’re looking for.

Likewise Software is in the identity management business. We make software that allows non-Windows systems to authenticate against Microsoft Active Directory and to employ AD-based group policy. As such, our needs at probably more sophisticated than those of most companies.

First, we need someone with good Windows networking/AD/DNS skills. Our biggest challenge at customer sites is assuring that their directories are properly configured. Our employees (especially our deployment engineers) need to be familiar with Active Directory and its architecture. They need to be able to run Likewise and Microsoft tools to assure that AD is properly configured and working properly. They need to be comfortable using tools like ADSIEDIT to look at objects in AD and they need to know what LDAP is. Experience with DNS and UNIX Bind is also valuable. Customers who choose to use Bind have to properly configure it to forward to AD/DNS or they have to manually set up a series of service records. Familiarity with NSLOOKUP and other tools is valuable.

Because random things can always go wrong when using a network, familiarity with network analyzers such as Ethereal is also valuable.

Second, we need someone with good Windows administrative skills. They have know how users are created in AD, how access controls are applied to resources and how Group Policy is used to help manage systems. They have to have some sense of how organizational units are used in AD and how GP objects can be inherited to accomplish company and departmental security and management goals.

Third, we need experience with UNIX and Linux administration. We support numerous versions of each so having familiarity with different shells and editors is a plus. You can’t rely on bash or vi being available on every system.  Different versions of UNIX and Linux also have their own vagaries regarding where they store certain files and how they start/stop daemons. Having rudimentary knowledge of different places to look/techniques to use is important. You might be working on HPUX one minute and on Ubuntu the next. Some knowledge of how local accounts are stored in /etc/passwd and /etc/shadow is a must.

Fourth, we need someone with rudimentary knowledge of UNIX/Linux architecture. Knowledge of PAM and NSSWITCH is valuable. Understanding how name resolution works and how networks and firewalls are configured is valuable, too.

Fifth, some cursory programming skills are useful. We frequently need to write or modify shell scripts to help with deployments or testing/monitoring tools. Our account migration tools can generate scripts and being able to modify those is also valuable.

Sixth, some Mac knowledge comes in handy as does some experience with Linux Gnome desktops.

Seventh,  general knowledge of Kerberos, Kerberos-based SSO and Kerberized applications is useful.

Finally, some experience with third party identity management systems is useful since we often need to interface with IBM ITIM or Sun Identity Manager or Microsoft ILM.

If you know anyone that meets all of these qualifications let me know. I’m pretty sure that I’ve hired all four of them and that my competitors have the other the other four. 🙂

Of course, we don’t expect candidates to have all of these skills. We’re lucky if they have half of them. My observation, however, is that our needs, if you factor out a couple of domain-specific things (e.g. Kerberos and LDAP), are not far from what any modern IT department needs if they’re running a heterogeneous data center. The amount of information that you need to know to effectively manage both Windows and non-Windows computers is huge. It’s not surprising that many departments choose to segregate these duties and assign them to different teams. As an unfortunate consequence, however, there is often little interaction and, sometimes, open hostility between these teams. Introducing interoperability solutions is complicated by the inherent distrust between the two camps. IT departments would do well to encourage education and personnel movement between the teams as a way to cross-pollinate ideas.