Now that it looks like it will really happen, I think I can post about it…

A little over a week ago, one of the directors where I work asked to talk to me. During the course of the discussion, he told me he wanted me to work primarily on sysadmin projects for his group. The reasoning? He told me that it was because I was “appropriately lazy”. He’d prefaced this statement with a disclaimer that what he was going to say was intended as a compliment. Upon hearing it, I assured him there was no way I could take the statement as anything other than a compliment. Here’s why…

A sysadmin’s day is often filled with lots of little tasks. Most could be unique for that day, some repetitve. In either case, you’ll find that, over time, you’re doing a lot of the same thing, day in and day out. Sometimes it doesn’t seem like it’s worth it to write a script or fully automate a task. After all, it only takes a minute or two, right?

Scripting

If it’s truly a one-off, yeah, it probably makes no sense to automate it in some fashion. If it’s something you’re doing on any recurring basis, though, it makes sense to at the very least write a script to take care of the details.

My general rule of thumb is this: “The first time I do something, I’ll tend to do it manually. If I have to do it a second time, I’ll take the time to either make it a wholly automated process, or at the very least, write a script to do the majority of the work for me.”

Why do I do this? Well, if it’s a one-off process, there’s no point in taking the time to make sure I can repeat something. If I’m doing it a second time, there’s a pretty good chance that I’ll be doing it a third, fourth, …, hundredth time. Worse, if it’s something I only do once a month or less frequently, there’s a really good chance that by the time I need to do it again, I’ll have forgotten the exact sequence of steps that I need to perform.

In either case, from my point of view it makes sense to write a script that does the majority of the work. Sure, it may take me a couple hours in the worst case to write a really good script to take care of what’s normally five minutes’ work, but when that’s 5 minutes each time, over five times a week–every week–for the next year or two, it’s worth the investment. When it’s something that will take me 20-30 minutes to figure out just what I did a month or two ago, spending a few hours to make sure I have a script that consistently and reliably does the same process is a “win” for me. It frees me from having to remember the details, and it ensures that I don’t forget some necessary step which seems optional, but in reality isn’t. ((This bit me recently, setting up a second VM for a customer. The first time, I found these exceptions and edge cases, such as adding the user tomcat was running as to the local adm group. I should have scripted it sooner, when I set up the first VM about a year ago…))

Package Management

Another area where I’m “appropriately lazy” is when it comes to package management. Whenever possible, I try to use my (Linux) distribution’s package management system for every software installation. True, this can often lead to slightly outdated software, but things like backports can help with that.

For software that isn’t available direct from my distribution of choice, there are often helper scripts to turn standard tarballs into packages, or often language-specific scripts to turn said language’s version of packages (CPAN archives, PHP packages, Ruby gems, etc) into native packages. Setting up a local repository to host these packages is fairly simple to do these days, and the benefits will out-weigh the time it takes to initially set things up.

For some packages, such as language-specific database drivers, ((such as Oracle drivers for PHP)) you’ll still find yourself compiling directly from source. However, if your distro is even slightly intelligent, it will have support for building these additional drivers when you rebuild the main package. You’ll probably have to uncomment some lines from a configuration file, but you’ll end up with another installable package that’s managed natively.

By sticking to native packages as much as possible, you’ll find yourself either adding an additional supported repository to each system (preferably in an automated fashion, such as with puppet), or building packages locally on a single build-host and adding to a local repository, which you’ll also add to your hosts. In either case, it means you’re, at most, building each piece of software a single time, and installing on multiple systems. This, obviously, will save you time and effort. You’ll only build once, and you’ll know that every place you install, the software will be built identically, because you’re installing the exact same software everywhere.

Meta-Packages

As a subset of building packages, there’s the topic of meta-packages. These don’t actually install any particular software themselves, they’re just a convenient way of installing other software.

As an example, I have meta-packages for installing everything I need for a webserver, with a sub-package for installing PHP with all the extensions our developers require. I also have meta-packages for installing the requirements for Java application servers, both tomcat and JBoss. Additional meta-packages I use are for LDAP authentication, “helpers” such as various utilities I’ve found useful, anti-virus scanners, file-access systems like FTP, perl modules, etc.

Having these meta-packages saves me both time and effort, by encapsulating in a single install everything I need to accomplish a specific task. Were I to have to manually install each necessary package on every server, there’s no doubt that I’d forget something, eventually causing a problem later down the line. ((This has happened, which is why I now use meta-packages…))

It also ensures there’s a consistency to each system I build. I, and my users, know that when I say “standard web build”, we have a known set of software we’re referring to. Whenever a user requests something generally useful for others, I’ll tend to add the requirement to one of my meta-packages. Installation will be automatic on all new systems, and will be taken care of on my weekly security updates for all other systems using that meta-package.

Regular Updates

Yes, I regularly update my systems. I attempt to do so once a week. This tends to keep me ahead on both security updates and (on the rare occasion) newer versions of the software we use. Generally, this also means that when our monthly security scans are run, I’m already patched for the latest vulnerabilities that our scanner has been updated to detect. Even when the software version isn’t modified, as is usually the case in a “stable” distribution, I at least have a changelog for security updates that list the CVE numbers of the problems fixed. This keeps our security groups (both local and at our parent) happy, because I can easily address any “problems” found in the scan, simply stating we’re already for a list of vulnerabilities from our “vendor”.

What that means is, I’m often not scrambling to update dozens or hundreds of systems under threat of being cut off at our firewall(s), since I’m very easily able to address concerns. This frees me up to concentrate on the other areas for which I’m responsible. Some of my colleagues will spend the better part of a week updating the systems they’re responsible for, often running into compatibility problems because it’s been so long since they last updated their systems. ((Yes, I’m preaching the benefits of regularly updating, using packages when possible, etc. I think I’m slowly winning them over.))

Currently, I’m using a tool to run ssh in parallel ((Update 2011-06-01 22:41 Someone’s asked which tool I use. It’s clusterssh, since I can define different classes, such as “webservers”, “tomcat servers”, “infrastructure”, etc, and access whole groups that way. It also has the ability to stop sending typed commands to specific windows it opens, so if I’ve already performed some steps to one server, I can avoid repeating steps that might not work correctly… like reconfiguring the mail daemon recently…)) to several systems at once. Updates are one of the areas I’m not entirely comfortable fully automating, just in case there are problems. But it’s an area where having all your systems configured from the same base image, with known variations to accomplish specific tasks, can significantly simply the amount of effort you need to expend keeping everything up to date.

Monitoring

I mostly touched upon this in a previous post, but it bears repeating (and linking). By setting up automated monitoring for everything I do, I’m often alerted to problems before anyone else notices. This allows me to fix problems before they become major problems, and if they do I can always immediately respond that I’m already aware of and (preferably) working on a resolution to the problem.

You’d be amazed how quickly this response keeps both customers and your management from breathing down your neck. You’ve demonstrated you already have a grasp of the situation, and that you’re (actively) working on the problem. Often times, in the interests of full disclosure, I’ll tell my customers and management about problems they didn’t notice, because I was alerted to them long before they would have noticed them & fixed them. There’s nothing like tooting your own horn sometimes…

Templates

A lot of the systems I deploy are virtual machines (VMs). Primarily, that’s because a lot of them are fairly identical. Also, since so many of the systems I deploy use a fraction of the resources of a modern machine ((Even our most heavily-used web servers, for example, could easily be served with the resources of a machine 4 or 5 years old.)), virtualization makes a great deal of sense from machine cost, cooling, and power requirements points of view. I’ll probably address the advantages of appropriate virtualization at some point in the future. For now, just accept that it can definitely be good.

Now, and advantage of most virtualization systems is that you’re able to define templates. Essentially, this means you set up and configure a minimal system, with all the common configuration you can. Then, you can deploy these templates to specific VMs. You save time, you have a known and consistent base you’re starting from, and you’re able, over time, to adapt it to your evolving needs. Needs such as incorporating your security updates, meta-package updates, and any overall changes you need to make.

Wrap-up

If you’ve read to this point, you’re probably thinking to yourself, “This doesn’t sound very lazy…“. You’d be right. The point of all of this isn’t to be lazy, the point is to be appropriately lazy. There is a difference. Being lazy, you or I wouldn’t bother to do most, if not all, of the things I’ve outlined. Being appropriately lazy, I (and now I hope, you) recognize that putting forth extra effort near the beginning will pay off multiple times over in expended effort later on. We recognize that taking a few hours (or even days) to make our future lives significantly easier is a good investment, not only in our current jobs, but in our overall careers. An appropriately lazy system administrator is a much more productive system administrator. ((Or programmer, or mathematician, or really, anyone… I did start my career as a programmer, and will probably at some point do such work again. What I’ve learned as a programmer is applicable as a systems administrator, and what I’ve learned as a sysadmin is applicable as a programmer…))

This is why, when I was told I was “appropriately lazy”, there was no way I could have taken it as anything but a compliment. Actually, it’s probably one of the highest compliments I could have been given. Especially, a compliment by a director in my organization who recognizes that visibly-expended effort isn’t nearly as valuable as visibly-saved effort over time.