Sysadmin Tales of Terror

By Carla Schroder | Feb 19, 2003 | Print this Page
http://www.enterprisenetworkingplanet.com/netsysm/article.php/1587441/Sysadmin-Tales-of-Terror.htm

The biggest challenge a system administrator ever faces is inheriting a networking mess: taking on a new job, or a new client, with a computing infrastructure that has grown without rhyme or reason. Lost passwords, little or no documentation, no-name network components, and cables snarled into alternate dimensions are just a sample of the evils to face and conquer. Then there are issues like multiple versions of operating systems and applications as well as strange odors coming from the electric panels. And you can't be sure the previous admins didn't leave a booby trap or two. Management is shellshocked and disinclined to hear more bad news -- or spend more money for that matter. What's the honest, hardworking sysadmin to do?

Plan A is run for your life. No one will blame you. But let's face it, you didn't come here to read that. I find it extremely satisfying to clean up nice big messes; it's a great challenge and a very rewarding one at that. Let's look at some strategies for dealing with such a situtation. While there are many possible approaches, these are some of the things that have worked for me.

Triage

The longer you stay on a job, the more icky stuff you'll find. Still, take the first few days on a new job or project to completely assess the situation and to get a better idea of how things currently work. I like to locate, in this order:
  • servers
  • firewalls
  • switches/hubs
  • routers
  • workstations
  • printers, scanners, misc. peripherals
While you're welcome to get all fancy and make Visio or Dia diagrams if you like, plain old graph paper and a pencil work fine for me. The important thing is that you document the network and start building an inventory. Ideally, you'll have an opportunity to focus on this before having to immediately react to new crises. It's better to map things out at the beginning as much as possible. Many problems and their solutions become obvious as you figure out how everything goes together -- and how everything should go together.

It's easy to get bogged down in fixing things that are substandard rather than focusing on the truly broken items. If something works, no matter how limpingly, assign it a lower priority than things that don't work at all.

Managing Expectations

This is the most important part of all: always "under promise and over deliver." Bosses often have no idea how messed up their computing infrastructure really is. To non-technical people it's all a magical blur. If management knew there was a nice big mess when you were hired, you're ahead of the game. If not, well, they need to know, and you're the one to deliver the sad news. Never ever badmouth your predecessors; just lay the situation out factually. This is a crucial juncture as you need management support to do what needs to be done. If they are not willing to listen to reason, or to heed your expert advice, then this is the time to bail. But ideally you'll be able to work together to set priorities and map out a feasible plan.

"Feasible" is the key word here. We would all love to work in the finest, state-of-the-art environments, using only the finest, most elegant code and studliest hardware. Real life is a bit messier than that. Again, it's triage and assigning priorities. There's never a perfect answer; just stay focused on the needs of the business as the basis for your decisions.

People skills are the #1 most important ability for the ace sysadmin. Our job is to serve the people we work with, not the machines. Technical ability is #2. I don't know how the image of the difficult, arrogant, unwashed geek became such a popular icon. In the real world, no one is so wonderful as to be excused from the usual social graces. Being likable, dependable, and pleasant to be around makes everything go better. Of course, this is not the same as being a pushover -- you have to do your job and be able to successfully advocate for your decisions and policies. But there is absolutely no good reason to ever be a jerk.

Page 2: Cover One's Behind With Glory

Cover One's Behind With Glory

Now let's be honest, documentation is boring and no fun. I don't care; just do it. Keep a project diary. Record everything you find. You don't want to shoulder the blame for someone else's mistakes or malfeasance. It is unlikely you'll get into legal trouble, but the possibility always exists. Record progress and milestones as well. Those in management tend to have short memories and limited attention spans when it comes to technical matters, so put everything in writing and make a point of reviewing your progress periodically. No need to put on long, windy presentations -- take ten minutes once a week to hit the high points. Emphasize the good news; after all, as the ace sysadmin, it is your job to make things work. Any dork can make a mess; it takes a real star to deliver the goods.

Be sure to couch your progress in terms meaningful to the person(s) you're talking to. A non-technical manager doesn't want to hear how many scripts you rewrote or how many routers you re-programmed. She wants to hear "Group A's email works flawlessly now, and I fixed their database server so it doesn't crash anymore. No more downtime for Group A." That kind of talk is music to a manager's ears.

Managing Users

In every business there are certain key people who wield great influence. They can make or break you. Don't focus exclusively on management -- the people who really run the show are the secretaries and administrative assistants. They know more than anyone about how things work, what's really important, and who is really important. Consult them. Listen to them. Suck up to them. Trust me, this will pay off handsomely. Also worth cultivating are relationships with the cleaning and maintenance people -- they see things no one else even knows about.

When you're new on the job and still figuring things out, the last thing you need is to field endless phone calls from users with problems. Make them put it in writing -- email, yellow pad, elaborate trouble-ticket system, whatever suits you. This gives you useful information and time to do some triage.

Managing Remote Users

If you have remote offices under your care, the phone can save a lot of travel. There's almost always one computer-savvy person in every office; make this person your ally and helper. At very least, this person will be able to give you coherent, understandable explanations. At best, they will be your remote hands and eyes, and will save you much trouble.

Such a person may be a candidate for training and possibly transferring to IT. Some people are afraid of helping someone like this for fear of losing out to them in some way. The truth, though, is that you never lose by helping people, so don't let that idea scare you off from giving a boost to a worthy person.

Getting Help

We all know how to use Google, Usenet, and other online resources to get assistance. By all means, don't be too proud -- ask! And by all means, don't be stupide either -- use a fake name and don't mention the company you work for. There's absolutely no upside to making such information public; there are, however, many downsides to doing so, like inviting security breaches, giving away too much information, making your company look bad, and besmirching your own reputation.

As I said at the beginning, these are strategies that have served me well. Feel free to send me your own ideas; I especially love to hear about true-life horror stories that have happy endings.

Resources

Life in the Trenches: A Sysadmin Speaks
10 Tips for Getting Along with People at Work
Linux Administration Books


» See All Articles by Columnist Carla Shroder