credit: “LAZY DOG” by paddynapper

A little over a week ago, one of the directors where I work asked to talk to me. During the course of the discussion, he told me he wanted me to work primarily on sysadmin projects for his group. The reasoning? He told me that it was because I was “appropriately lazy”. He’d prefaced this statement with a disclaimer that what he was going to say was intended as a compliment. Upon hearing it, I assured him there was no way I could take the statement as anything other than a compliment. Here’s why…

A sysadmin’s day is often filled with lots of little tasks. Most could be unique for that day, some repetitve. In either case, you’ll find that, over time, you’re doing a lot of the same thing, day in and day out. Sometimes it doesn’t seem like it’s worth it to write a script or fully automate a task. After all, it only takes a minute or two, right?

Scripting

If it’s truly a one-off, yeah, it probably makes no sense to automate it in some fashion. If it’s something you’re doing on any recurring basis, though, it makes sense to at the very least write a script to take care of the details.

My general rule of thumb is this: “The first time I do something, I’ll tend to do it manually. If I have to do it a second time, I’ll take the time to either make it a wholly automated process, or at the very least, write a script to do the majority of the work for me.”

Why do I do this? Well, if it’s a one-off process, there’s no point in taking the time to make sure I can repeat something. If I’m doing it a second time, there’s a pretty good chance that I’ll be doing it a third, fourth, …, hundredth time. Worse, if it’s something I only do once a month or less frequently, there’s a really good chance that by the time I need to do it again, I’ll have forgotten the exact sequence of steps that I need to perform.

In either case, from my point of view it makes sense to write a script that does the majority of the work. Sure, it may take me a couple hours in the worst case to write a really good script to take care of what’s normally five minutes’ work, but when that’s 5 minutes each time, over five times a week–every week–for the next year or two, it’s worth the investment. When it’s something that will take me 20-30 minutes to figure out just what I did a month or two ago, spending a few hours to make sure I have a script that consistently and reliably does the same process is a “win” for me. It frees me from having to remember the details, and it ensures that I don’t forget some necessary step which seems optional, but in reality isn’t.¹

Package Management

Another area where I’m “appropriately lazy” is when it comes to package management. Whenever possible, I try to use my (Linux) distribution’s package management system for every software installation. True, this can often lead to slightly outdated software, but things like backports can help with that.

For software that isn’t available direct from my distribution of choice, there are often helper scripts to turn standard tarballs into packages, or often language-specific scripts to turn said language’s version of packages (CPAN archives, PHP packages, Ruby gems, etc) into native packages. Setting up a local repository to host these packages is fairly simple to do these days, and the benefits will out-weigh the time it takes to initially set things up.

For some packages, such as language-specific database drivers,² you’ll still find yourself compiling directly from source. However, if your distro is even slightly intelligent, it will have support for building these additional drivers when you rebuild the main package. You’ll probably have to uncomment some lines from a configuration file, but you’ll end up with another installable package that’s managed natively.

By sticking to native packages as much as possible, you’ll find yourself either adding an additional supported repository to each system (preferably in an automated fashion, such as with puppet), or building packages locally on a single build-host and adding to a local repository, which you’ll also add to your hosts. In either case, it means you’re, at most, building each piece of software a single time, and installing on multiple systems. This, obviously, will save you time and effort. You’ll only build once, and you’ll know that every place you install, the software will be built identically, because you’re installing the exact same software everywhere.

Meta-Packages

As a subset of building packages, there’s the topic of meta-packages. These don’t actually install any particular software themselves, they’re just a convenient way of installing other software.

As an example, I have meta-packages for installing everything I need for a webserver, with a sub-package for installing PHP with all the extensions our developers require. I also have meta-packages for installing the requirements for Java application servers, both tomcat and JBoss. Additional meta-packages I use are for LDAP authentication, “helpers” such as various utilities I’ve found useful, anti-virus scanners, file-access systems like FTP, perl modules, etc.

Having these meta-packages saves me both time and effort, by encapsulating in a single install everything I need to accomplish a specific task. Were I to have to manually install each necessary package on every server, there’s no doubt that I’d forget something, eventually causing a problem later down the line.³

It also ensures there’s a consistency to each system I build. I, and my users, know that when I say “standard web build”, we have a known set of software we’re referring to. Whenever a user requests something generally useful for others, I’ll tend to add the requirement to one of my meta-packages. Installation will be automatic on all new systems, and will be taken care of on my weekly security updates for all other systems using that meta-package.

Regular Updates

Yes, I regularly update my systems. I attempt to do so once a week. This tends to keep me ahead on both security updates and (on the rare occasion) newer versions of the software we use. Generally, this also means that when our monthly security scans are run, I’m already patched for the latest vulnerabilities that our scanner has been updated to detect. Even when the software version isn’t modified, as is usually the case in a “stable” distribution, I at least have a changelog for security updates that list the CVE numbers of the problems fixed. This keeps our security groups (both local and at our parent) happy, because I can easily address any “problems” found in the scan, simply stating we’re already for a list of vulnerabilities from our “vendor”.

What that means is, I’m often not scrambling to update dozens or hundreds of systems under threat of being cut off at our firewall(s), since I’m very easily able to address concerns. This frees me up to concentrate on the other areas for which I’m responsible. Some of my colleagues will spend the better part of a week updating the systems they’re responsible for, often running into compatibility problems because it’s been so long since they last updated their systems.⁴

Currently, I’m using a tool to run ssh in parallel⁵ to several systems at once. Updates are one of the areas I’m not entirely comfortable fully automating, just in case there are problems. But it’s an area where having all your systems configured from the same base image, with known variations to accomplish specific tasks, can significantly simply the amount of effort you need to expend keeping everything up to date.

Monitoring

I mostly touched upon this in a previous post, but it bears repeating (and linking). By setting up automated monitoring for everything I do, I’m often alerted to problems before anyone else notices. This allows me to fix problems before they become major problems, and if they do I can always immediately respond that I’m already aware of and (preferably) working on a resolution to the problem.

You’d be amazed how quickly this response keeps both customers and your management from breathing down your neck. You’ve demonstrated you already have a grasp of the situation, and that you’re (actively) working on the problem. Often times, in the interests of full disclosure, I’ll tell my customers and management about problems they didn’t notice, because I was alerted to them long before they would have noticed them & fixed them. There’s nothing like tooting your own horn sometimes…

Templates

A lot of the systems I deploy are virtual machines (VMs). Primarily, that’s because a lot of them are fairly identical. Also, since so many of the systems I deploy use a fraction of the resources of a modern machine⁶, virtualization makes a great deal of sense from machine cost, cooling, and power requirements points of view. I’ll probably address the advantages of appropriate virtualization at some point in the future. For now, just accept that it can definitely be good.

Now, and advantage of most virtualization systems is that you’re able to define templates. Essentially, this means you set up and configure a minimal system, with all the common configuration you can. Then, you can deploy these templates to specific VMs. You save time, you have a known and consistent base you’re starting from, and you’re able, over time, to adapt it to your evolving needs. Needs such as incorporating your security updates, meta-package updates, and any overall changes you need to make.

Wrap-up

If you’ve read to this point, you’re probably thinking to yourself, “This doesn’t sound very lazy…“. You’d be right. The point of all of this isn’t to be lazy, the point is to be appropriately lazy. There is a difference. Being lazy, you or I wouldn’t bother to do most, if not all, of the things I’ve outlined. Being appropriately lazy, I (and now I hope, you) recognize that putting forth extra effort near the beginning will pay off multiple times over in expended effort later on. We recognize that taking a few hours (or even days) to make our future lives significantly easier is a good investment, not only in our current jobs, but in our overall careers. An appropriately lazy system administrator is a much more productive system administrator.⁷

This is why, when I was told I was “appropriately lazy”, there was no way I could have taken it as anything but a compliment. Actually, it’s probably one of the highest compliments I could have been given. Especially, a compliment by a director in my organization who recognizes that visibly-expended effort isn’t nearly as valuable as visibly-saved effort over time.

1. This bit me recently, setting up a second VM for a customer. The first time, I found these exceptions and edge cases, such as adding the user tomcat was running as to the local adm group. I should have scripted it sooner, when I set up the first VM about a year ago…
2. such as Oracle drivers for PHP
3. This has happened, which is why I now use meta-packages…
4. Yes, I’m preaching the benefits of regularly updating, using packages when possible, etc. I think I’m slowly winning them over.
5. Update 2011-06-01 22:41 Someone’s asked which tool I use. It’s clusterssh, since I can define different classes, such as “webservers”, “tomcat servers”, “infrastructure”, etc, and access whole groups that way. It also has the ability to stop sending typed commands to specific windows it opens, so if I’ve already performed some steps to one server, I can avoid repeating steps that might not work correctly… like reconfiguring the mail daemon recently…
6. Even our most heavily-used web servers, for example, could easily be served with the resources of a machine 4 or 5 years old.
7. Or programmer, or mathematician, or really, anyone… I did start my career as a programmer, and will probably at some point do such work again. What I’ve learned as a programmer is applicable as a systems administrator, and what I’ve learned as a sysadmin is applicable as a programmer…

Another push I’ve been working on for a few years now is comprehensive monitoring of our systems. When I started at my job, monitoring was being done through a home-grown script, that would run once an hour. While not particularly bad, it had some major shortcomings. Primarily, the hour lag time meant that, if a problem occurred shortly after the script ran, you wouldn’t know about it for nearly an hour. Another problem was that, while the script would make note of system problems, the only notification mechanism was “someone needs to visit the web page it generates”. During a normal workday, that might be OK, but evenings and weekends? So, at least for my benefit, I set up a monitoring system that could not only monitor systems more frequently, but monitor more systems, and send out alerts via email, cell phones, etc.

For reasons I won’t get into here, what I chose to use for configuration management was Puppet, and I’d already chosen Nagios as my monitoring system. So far, my biggest problem with Nagios has been finding the time to add new systems to it, figuring out what services to check, etc. It’s not a particularly difficult thing to do, but in the grand scheme of things, it was just something that always fell by the wayside in the drive to get more systems set up, deal with user problems, and put out the inevitable fires. That is, until recently.

My original Nagios install was done on an aging box that has been serving as a development web server for our group. I did the install shortly after I’d started, so had I been around longer, I might have either waited, put in a request for new hardware, or found a way to get virtualization into our environment sooner. Overall, it worked, but it was in general a pain dealing with the box, a RHEL3 server, since it was so far out of date. Recently, I’ve set up a VM on our XenServer cluster to be our new Nagios box. Since I’ve also been playing with Puppet, I wanted to automate things as much as possible, since every new system created should be managed by Puppet (though reality slightly differs from that).

Fortunately, the groundwork for automating Nagios monitoring with Puppet is already built in to Puppet. It took me a little bit to wrap my head around the concepts, but the example helped and served as a base.

Now, even though I’m still in the process of setting things up, I’ve gotten to the point where my new Nagios server is already monitoring about 30% more information about hosts (422 services vs 308 on the old server), even though the number of hosts is currently about half (48 hosts vs 99 on the old). All of it done automatically, and here’s how.

First, set up stored configurations on your puppetmaster. You’ll need to specify a database in which to store your puppet-collected facts and resources. While the default is SQLite, I ran into problems with concurrent access. Since I’m also currently responsible for the handful of MySQL servers we have, I decided to just use one of those. Create a database and user for puppet to use, then tell the puppetmaster about it. Your [puppetmasterd] section of puppet.conf should look something like this when you’re done:

Your paths may likely be different than mine. If your DB server is running on the same host as the puppetmasterd process, dbserver should be “localhost”, and you’d uncomment and adjust the path of the dbsocket line. The downcasefacts line is set to “true” so that I can make use of the $operatingsystem fact later on without having to muck with changing the case later.

Next, you’ll want to create a nagios module in Puppet. The Exported Resource example linked above served as my template, but I’ve made a few changes to it. My puppet/modules/nagios/manifests/init.pp file currently looks like this:

To use it, I simply do an include nagios in the node definition for my Nagios server in puppet, and in my basenode node definition, I’ve done an include nagios::target. Each of the @@ lines will collect information for each machine managed by puppet that inherits from basenode. The “collect resources and populate /etc/nagios/nagios_*.cfg” portion is the real magic, however. Each of those lines will cause puppet to collect all the matching resources, and output them to files in /etc/nagios. The only real caveat, which I also noticed in the example I built upon, is that I’m having trouble convincing puppet to reload nagios when the files are updated, which I just brute-force solved with a periodic cronjob to run nagios’ init script with “reload”.

I’m also slowly adding nagios entries for each service that puppet manages in some form. Currently, that means things like apache and ssh. For example, in my apache2 module’s init.pp, I have the following in my class:

This both monitors over-the-wire connections to port 80 on webservers, via the check-http command, but also monitors the number of httpd processes running on each host, via remote-nrpe-httpd-procs.

Similarly, for ssh, I have:

to monitor whether sshd is accepting connections on my systems.

And that, basically, is how I’m automatically monitoring all puppet-managed hosts in my environment. Whenever I set up a new host, I activate puppet on the host to ensure configurations I care about are synced to my master templates, and now as a bonus, puppet automatically tells nagios to start monitoring the services it knows about on the host. By expending a little extra effort once now, I’ve managed to be lazy later on multiple times over, truly something a Systems Administrator should be doing!