Editor’s Note: When he’s not busy clarifying networking concepts, Enterprise Networking Planet’s Charlie Schluting spends some of his spare cycles on Ourmon, an open source network security monitor developed at Portland State University in Oregon. Here’s the second of two pieces from Charlie on how Ourmon works and how to set it up.
Last week we introduced Ourmon, the network monitoring and anomaly detection system, and explained how it can help you understand your network usage and detect infected machines. Let’s take a look at how it all works.
From an architectural point of view, the Ourmon probe needs to be located on a switch that sees all packets before they enter/leave your site. The idea is to mirror all data from the port(s) that are external links, to send a copy of every packet to the Ourmon probe. Of course, you can also install Ourmon on a single workstation with a standard connection, to monitor just that host.
Ourmon works best on FreeBSD, but also can be used on Linux and Solaris machines. If you’re going to monitor a heavy load (more than 100Mb/s) then FreeBSD is recommended. To install it in FreeBSD, you can simply use the ports system:
cd /usr/ports/net-mgmt/ourmon && make install
For Linux installs, you’d grab the tarball from the Web site, untar it in the final install directory of your choice, and then run configure.pl to configure and install it. The FreeBSD port is basically doing the same thing, so the procedure is identical from this point on.
When you run the configure.pl script, it will begin to ask you some questions. You can accept the defaults, but pay attention to the paths it asks about, so you know where the web and log files end up. You can also set up Ourmon to live on separate machines, to farm out the back-end processing. You’ll want to read the install instructions for setups like this. In fact, it isn’t a bad idea to read them anyway.
Ourmon will install cron jobs to run frequently, if you chose to install everything on the same machine. As soon as the install process is complete, you’ll want to follow the instructions and run the ourmon.sh script: ‘ourmon.sh start’ to avoid cron errors.
You will find, in the etc/ directory where Ourmon was installed, an ourmon.conf file. This is your vehicle for creating new graphs and filters. Creating new graphs is done by simply creating a new BPF filter in ourmon.conf, using the examples as a guide. Next, you must go to the Web directory and edit index.html to include the new graph image. It’s not too much trouble, and the ourmon.conf and INSTALL files explain exactly how to do this. Do pay attention to the suggestion about testing out BPF expressions with tcpdump. Unless you’re extremely familiar with BPF, you’ll most likely get it wrong the first time.
So what can it do? The installation documents give a few examples, and they cover the most common need for adding new graphs: subnet graphs. It is very nice to separate out traffic per subnet, so that you can trace traffic spikes and other anomalies quicker. There are other examples given, and the graphs are only limited by your imagination and tcpdump chops.
After Ourmon has run for a bit, you’ll notice the graphs coming to life. After Ourmon has run for a few days, you’ll start to see what traffic on your network normally looks like, assuming you aren’t under attack at the moment. Nothing will start to stand out, until you have an idea of what your baseline really is. And then some day you’ll come in and see a huge spike in packets per second, bytes per second, or both.
(Click for a larger image)
A huge spike in traffic may correspond to a huge spike in the IRC report graph too, and if that happens you’re either massively infected or running a botnet server. Take, for example, figure 1, gathered from PSU not too long ago.
This is the count of TCP SYN packets per period. Clearly something is happening. It turned out that an unattended Windows computer had been taken over, and started running a bot server.
If you were to scroll down the page and notice a big spike in the IRC count graph as well, you’d readily know what it was. Figure 2 is a weekly view of the IRC count graph.
(Click for a larger image)
IRC implements a layer 7 ping/pong mechanism to detect when clients have disappeared. If your graph all of a sudden sees 2000 pongs, when about 10 per period is normal, that likely means you’re running an IRC server with thousands of clients. Even on a network with a few legitimate IRC servers, of average popularity (a few hundred clients), there is nowhere close to 2000 ping or pong messages.
The next step is to look at the “top SYNner” report to find the IP of the machine causing this, and then to go unplug the machine. It all comes down to knowing what the norm is on your network. The nature of RRD graphs helps in this regard; you can go back, up to a year, to see traffic data.
Ourmon graphs will show external traffic too. If someone is scanning a bunch of machines on your network, that will show up in the TCP SYN graphs. Some graphs have a concept of “us versus them,” but not all. You can set up graphs similar to the live demo page to graph “us” and “them” in the same place. These types of graphs provide an excellent overview of who is sending traffic when you’re investigating an anomaly.
This is barely scratching the surface of Ourmon’s capabilities. All in all, Ourmon provides many things that network engineers may currently be without: network traffic graphs, at the granularity of layer 4 protocols (or whatever you concoct); the ability to segment network traffic graphs based on subnet (or anything else); and of course, anomaly detection, something snort can’t even claim. Even with the default configuration, Ourmon is indispensable.