Sysadmin Tales of Terror

The biggest challenge a system administrator ever faces is inheriting a networking mess: taking on a new job, or a new client, with a computing infrastructure that has grown without rhyme or reason. Lost passwords, little or no documentation, no-name network components, and cables snarled into alternate dimensions are just a sample of the evils to face and conquer. Then there are issues like multiple versions of operating systems and applications as well as strange odors coming from the electric panels. And you can't be sure the previous admins didn't leave a booby trap or two. Management is shellshocked and disinclined to hear more bad news -- or spend more money for that matter. What's the honest, hardworking sysadmin to do?

Plan A is run for your life. No one will blame you. But let's face it, you didn't come here to read that. I find it extremely satisfying to clean up nice big messes; it's a great challenge and a very rewarding one at that. Let's look at some strategies for dealing with such a situtation. While there are many possible approaches, these are some of the things that have worked for me.

Triage

The longer you stay on a job, the more icky stuff you'll find. Still, take the first few days on a new job or project to completely assess the situation and to get a better idea of how things currently work. I like to locate, in this order:

• servers
• firewalls
• switches/hubs
• routers
• workstations
• printers, scanners, misc. peripherals

While you're welcome to get all fancy and make Visio or Dia diagrams if you like, plain old graph paper and a pencil work fine for me. The important thing is that you document the network and start building an inventory. Ideally, you'll have an opportunity to focus on this before having to immediately react to new crises. It's better to map things out at the beginning as much as possible. Many problems and their solutions become obvious as you figure out how everything goes together -- and how everything should go together.

It's easy to get bogged down in fixing things that are substandard rather than focusing on the truly broken items. If something works, no matter how limpingly, assign it a lower priority than things that don't work at all.

Managing Expectations

This is the most important part of all: always "under promise and over deliver." Bosses often have no idea how messed up their computing infrastructure really is. To non-technical people it's all a magical blur. If management knew there was a nice big mess when you were hired, you're ahead of the game. If not, well, they need to know, and you're the one to deliver the sad news. Never ever badmouth your predecessors; just lay the situation out factually. This is a crucial juncture as you need management support to do what needs to be done. If they are not willing to listen to reason, or to heed your expert advice, then this is the time to bail. But ideally you'll be able to work together to set priorities and map out a feasible plan.

"Feasible" is the key word here. We would all love to work in the finest, state-of-the-art environments, using only the finest, most elegant code and studliest hardware. Real life is a bit messier than that. Again, it's triage and assigning priorities. There's never a perfect answer; just stay focused on the needs of the business as the basis for your decisions.

People skills are the #1 most important ability for the ace sysadmin. Our job is to serve the people we work with, not the machines. Technical ability is #2. I don't know how the image of the difficult, arrogant, unwashed geek became such a popular icon. In the real world, no one is so wonderful as to be excused from the usual social graces. Being likable, dependable, and pleasant to be around makes everything go better. Of course, this is not the same as being a pushover -- you have to do your job and be able to successfully advocate for your decisions and policies. But there is absolutely no good reason to ever be a jerk.

Page 2: Cover One's Behind With Glory