“We are not only downsizing but we are also stupefying when we lay off the
senior network engineers.”
So said an anonymous executive at a Fortune 100
company. Unfortunately, that means that you are now expected to handle
sophisticated network analysis issues in addition to the daily grind of
keeping the network running smoothly.
This has never happened to you. You know that your coworkers are using
instant messages to their friends. They are using their private accounts,
but your management has just asked you to put a stop to it. They allege it
is costing the company productivity and bandwidth. How can you find out if
their claims are true? In addition, what can you do about it now?
Your manager has told you to determine why the network connection to your
branch office in Kalamazoo runs so slowly. Is there a network traffic jam
or is something else causing the problem? Nobody else has complained about
the issue. When the network has problems, do you have the tools to detect
them so you can respond appropriately?
As the company security officer (in addition to all of your other hats), a
law enforcement agency has approached you. Someone was attempting to crack
the Pentagon computer systems using your company’s network. They would
like you to help track down who it was. Can you do anything? Is it
possible that your network was hijacked and you did not know it?
Welcome to the world of network and systems analysis tools. With a good
network and systems analyzer, you can know exactly what traffic is on your
network and precisely how your applications and systems are used. You can
also use monitoring tools to solve network problems in real time – critical
in today’s fast-paced 7×24 enterprises. The newer network analysis tools
help you solve your network and systems headaches and can guide your
business’s network and application growth but they are sophisticated tools
that take some training to understand and use properly.
Monitoring or analysis – What is the difference?
You are responsible for maintaining your company’s network and systems
infrastructure; you know that the days of using the old standbys, ping and
traceroute, to manage your infrastructure are long past. “Even a typical
mid-sized company network deployment with a 24 port 10/100 router box and a
couple of uplinks is going to generate an enormous amount of traffic data.
You are dealing with a network that potentially has 100MB to multiple
gigabits of data streaming through it. Saving the raw data for the amount
of traffic even in a small network and analyzing it by hand is a mind
boggling exercise, even if there was sufficient storage capacity or the
expertise to figure out the traffic data,” says Debra Deutsch, well-known
Network Industry Technologist, and a member of the BBN team that helped
create the Internet.
Of the huge variety of tools on the market today, which ones make your life
easier and which are not worth the effort? First, we need to define what
these tools are and how they are used. Network utilities fall into two
basic categories: real time monitoring, and systems analysis tools.
Real time monitoring tools are used for detecting events or problems on
your network in real time, i.e. as the faults are happening. These tools
are typically used by ISPs and enterprises to detect hardware failures,
hung systems, and other fatal infrastructure problems. Monitoring tools
can be simple or sophisticated, hardware based or software only, but they
all tend to be simple to access and use. When you have a bad network link,
you do not have time to spend on complex analysis or complicated tools, you
want to get your systems back on line quickly. A packet sniffer is a good
example — it tells you what traffic is on your network and where it is
coming from. If your company is large enough to have a staffed Network
Operations Center (NOC) you would want a powerful graphical user interface
(GUI) monitoring system.
The advantage to these tools is that they commonly use alarms to
instantaneously alert you to problems as they happen. Nevertheless, warns
John Eldridge, network engineer and principal at E-Bryonics, “You must be
prepared to support very large amounts of performance data for real time
access for troubleshooting efforts. The more network components deployed
and monitored, the larger the amount of data to manage.” Another
disadvantage is that they generally do not keep a history of what was
happening during the problem detection. This information is critical for
proactive network maintenance and prevention activities.
If all you need is a tool to tell you if your network is down and where
the fault is likely to be located, then a monitoring system is probably the
right tool for your needs. Monitoring tools available on the market today
range from a large number of shareware utilities like Big Brother or mon,
to powerful commercial products like Ciscoworks or the HP OpenView toolkit.
Most companies do not have the resources or the scale to justify the
expense of maintaining a large network monitoring staff, so you will
doubtless be interested in one of the more modest tools. If you are the
entire IT department, you would probably prefer to have the system page you
only when it detects a real problem. Who wants to be disturbed in the
middle of the night just to find out that the mail server rebooted?
Although there are many to choose from here are a few particularly useful
shareware programs. Big Brother is a set of local clients that test system
conditions and the availability of network services. It then sends
periodic status reports to one or more DISPLAY servers, or PAGER servers
that notify administrators about system problems. Big Brother can be a
very useful shareware utility for a LAN network site. It is simple to
install and easy to use. The mon website reports, “mon is a general purpose
scheduler and alert management tool used for monitoring service
availability and triggering alerts upon failure detection. mon was
designed to be open and extensible in the sense that it supports arbitrary
monitoring facilities and alert methods via a common interface, all of
which are easily implemented with programs in C, Perl, shell, etc., SNMP
traps, and special mon traps.” While mon is has many great features and a
powerful set of extensions, it will clearly work best for the expert
When the facilities design engineers at a GTE site in Irving started
complaining that one of their applications was running unacceptably slow,
they immediately blamed it on a LAN problem. They wanted to upgrade all
their workstations to 100MB Ethernet speed at great company expense. After
using some sophisticated infrastructure analysis tools to pinpoint the
issue, it was determined that they were attempting to run a large CADD
database application in real-time on a server located 1500 miles away.
Yes, there were latency problems related to the WAN, but the real cause of
the poor performance was that every time someone typed even one character
the system would attempt to update the file on the remote server.
Upgrading their network connections would have been useless. The ultimate
solution was to create a Citrix server on the user community LAN to
minimize the number of tiny packets sent halfway across the country!
You want to purchase a new network edge router, but management requirements
justification for the expense. Your users are complaining about the slow
network connection to the West Coast office. How do get the infrastructure
information you need to do your job? Using network systems analysis tools,
you can gain a better understanding of your network and systems so you are
able to answer these questions and solve enterprise systems problems. An
entirely new generation of tools is available today, which is an order of
magnitude more powerful and easier to use, but they are not for the
Analysis tools pick up where the monitoring tools leave off. They
generally work by filtering through the logs of captured systems data and
giving you a picture of possible problems from a historical perspective.
“One can use filters to select for specific destinations or sources at the
MAC, IP, UDP, or TCP Port numbers to cut down on the analysis processing.
Also, tools such as port filtering can be used to mirror in real time the
traffic that passes the filters,” comments Deutsch. Although analysis
tools can be fundamentally more powerful, they also are frequently more
difficult to use because you need to understand the results of the
analysis. How useful would the tool be if it informed you of a potential
flapping route and you do not know what it was and how to fix it?
Again, analysis tools come in a variety of flavors for all levels of skill
and sophistication. Bprobe & cprobe provide measurement of bottleneck and
congestion bandwidth using ping. These would be useful if you want to find
out if you have enough bandwidth or your users are just complaining on
general principles. On the high end of the spectrum is a tool like, Route
Explorer by Packet Design CNS, Inc. that lets you visually diagnose and
analyze your WAN IP routes over a long period.
Most analysis tools assume that the user would recognize the fault or
problem in the data flow by its pattern and characteristics. As a long
time user of these analysis systems, Eldridge notes, “In automating the
network analysis process, each network component needs to generate specific
SNMP MIB data to the network monitoring/management system. Adding the
intelligence to the tool to interpret the network statistics is a critical
step in improving the value of these systems to assist in identifying
performance issues. Statistics often differ depending on the network
elements being examined.”
Some gotchas to watch for
Unfortunately none of these tools are truly “plug and play”, so there is a
level of skill required to use them properly which can be daunting to the
technologically unsophisticated. For proper analysis, accurate data needs
to be gathered and must be appropriately organized. “In analyzing network
performance issues, a critical step is the creation of an accurate
inventory of the various network components of routers, switches, circuits,
etc. in order to identify any network component which is causing a
performance problem. This inventory must be integrated into any automated
analysis tool to increase its troubleshooting value,” says Eldridge.
Deutsch notes, “An important consideration is where on the network you are
watching the traffic. If you were doing real time analysis, you would
watch at the edge of your network where your traffic goes onto the
Internet. That could detect external nefarious activity (i.e. attacking
the Pentagon). Remember, not all hacking is directed at external targets,
so to detect internal hackers, it may be necessary to monitor multiple
internal network nodes as well.”
There are many elements that need to be taken into consideration when you
are analyzing a network for faults, the hardware, the network topography
itself, the systems and of course the applications. “A lot of this
information is dependent on your switch’s ability to handle the data flow
to allow you to analysis your traffic. Depending on what you are looking
for you might need to start at globally and refine your search as you
pinpoint your problem. Important is the ease in which you can on the fly
ability to refine what data you are capturing and filtering,” Deutsch
Although the available tools may be excellent, the art of infrastructure
analysis is still dependent on the skill and knowledge of the operators.
If you are managing your company networks, the best investment you can make
is some training in network architecture and analysis. So how do you learn
to use the available network analysis tools?
Fortunately, most networking engineering training programs include network
analysis modules. Cisco’s extensive and comprehensive offerings include a
number of good classes on their tools, so does HP for HP OpenView. If you
want to learn more about networks from a more theoretical perspective, I
would check if your local college or university has either a certification
program or just some network engineering classes. The education you will
get will be invaluable for your present job, as well as enhance your future
Tools and more tools
From what I saw at the last “Network world + InterOp” show, there are literally hundreds of tools out on the market to help you accomplish this
task. Which tools make sense for your IT infrastructure?
At the most basic level you need to decide if you want to look at your
systems in real-time or analyze the data after the events. Most companies
deploy some combination of both. Which tools you choose is also dependent
on the size of your company, the strategic value of your computer resources
and the company network topology. For example, every hour that Amazon.com
is unavailable to its customers can mean literally millions of dollars of
lost revenue. It is critical that Amazon have superb real-time monitoring
and excellent preventive maintenance tools. For Amazon, these tools and
the staff who know how to use them are strategic for business success. For
the majority of companies, something more modest is probably sufficient.
The tools available range from ISP quality advanced analysis software to
handheld wireless packet analyzers and everything in between. Here is just
a small sample of the available tools that I found particularly
Handheld Wireless Analyzer
Many companies are deploying 802.11 and 802.1x wireless networks because of
their lower cost and increased flexibility. AirMagnet Inc., a Mountain
View, CA based startup, offers a wireless analyzer with a built in
comprehensive suite of wireless troubleshooting tools in a palm-sized
Pocket PC. Its robust set of tools quickly helps eliminate connection
problems, maintain network performance levels, and ensure a high level of
network security. If you have a wireless network, this handy tool is
Visual IP Route Diagnostic Tool
Are you a visually oriented person? Would you like to see your network
traffic as a map rather than a text log? Route Explorer by Packet Design
CNS, Inc. lets you visually diagnose and analyze your WAN IP routes. You
can save up to a year of historical data about your network routes, and
create animations of the traffic flow patterns. Many of the big ISPs and
telecoms have had similar proprietary tools showing network node maps and
histograms for years. This tool allows you to do “what if” scenarios to
reduce the risk of network configuration errors, a major problem for many
corporations. Although I found the product appealing, it does require that
you have an advanced level of knowledge of WAN networking to take full
advantage of the features.
Network Flow Analyzer
The most comprehensive suite of analysis products is AppDancer/FA from
AppDancer Inc. in Roswell, GA. Their “Network Flow Analyzer” takes a fully
integrated approach to infrastructure analysis. By providing fully
threaded session analysis, this amazing tool not only monitors your
devices, applications and all associated IP flows, but it gives you a view
into the real inter-workings of your network. “Because it takes an
application centric view of the network, it can help you answer such
questions as “‘Is it the applications themselves or the demands or
limitations on the devices in my Network?’ or ‘Who is running an
unauthorized application using my network?'” according to Tim O’Neill,
AppDancer, Director of Sales. This tool does it all! For a mid-sized
company that needs sophisticated analysis but has minimal staff resources,
this tool could be very useful.
The new network and systems analysis tools can make your life as a system
administer much easier. Monitoring tools can quickly notify you of systems
and network problems, but they will not give you the history and analysis
that you need to prevent future problems. If you willing to take the time
to learn, some of the sophisticated analysis tools available today will
help you deliver a more reliable and robust network to your customers.
http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html#nmp-tool – Very
comprehensive website with information about Network (both LAN and WAN)
Monitoring tools designed for the network administrator.
http://ciac.llnl.gov/ciac/ToolsUnixNetMon.html – Website with pointers to
some useful UNIX monitoring tools
http://www.alw.nih.gov/Security/prog-network.html – More comprehensive
government site with pointers to a variety of utilities and tools.
Beth Cohen is president of Luth Computer Specialists, Inc., a consulting
practice specializing in IT infrastructure for smaller companies. She has
been in the trenches supporting company IT infrastructure for over 20 years
in a number of different fields including architecture, construction,
engineering, software, telecommunications, and research. She is currently
writing a book about IT for the small enterprise and pursuing an
Information Age MBA from Bentley College.