Tips and Tricks for Linux Admins: Keep an Eye On Your Servers
Today's networking grab-bag contains iftop, ApacheTop, and sysctl. iftop is a nice realtime bandwidth monitor, ApacheTop is an almost real-time Apache monitor, and sysctl is used to control hundreds of kernel parameters in a most elite fashion. sysctl comes with all Linux distributions, and iftop and ApacheTop are just a Yum install or apt-get install away.
With all the skillions of good network monitors, why should we care about yet one more? Because iftop is a small, easy-to-use monitor for watching a single interface. It shows information in a netstat-type format, plus bar graphs that show your line speeds. Suppose you're working in your living room on your wireless laptop, happily Web-surfing, and suddenly everything slows to a crawl. Fire up iftop to see what is happening:
# iftop -i ath0
If you're curious about what sort of line speeds your DSL or cable connection is delivering, iftop will tell you. Hit the lowercase t key to toggle different views, such as Sent traffic only, Received traffic only, and one or two-line views. h toggles the Help screen on and off, q exits the program.
n toggles name resolution, p toggles port resolution, and -p puts it in promiscuous mode, which is useful on a gateway machine. It's always instructive to visit a commercial Web site just to see how many adservers are pummeling your connection.
man iftop tells all options.
Old-fashioned gnarly geekbeards run tail -f /var/log/apache2/access.log to monitor their Apache servers, and it works fine thankyouverymuch. Reading raw logs has its own special charm, I suppose; I like how Apachetop organizes and aggregates the data in a nice sensible display, like the top command. Use it like this:
$ apachetop -f /var/log/apache2/access.log
To watch multiple files all lumped together, list them like this:
$ apachetop -f /var/log/apache2/access.log -f /var/log/apache2/error.log
It isn't real-time, but close enough, as Apachetop has to wait until entries are written to the logfiles. The -l option is very useful, as it treats all URLs as lowercase. So it won't count the same URL, for example carla.com and CARLA.com, as two different URLs. Use the -r option to set the refresh rate. The default is five seconds. CTRL+C exits. man apachetop lists all of the not-very-many command options.
A common way for us l33t admins to interact with the Linux kernel is with the sysctl command. sysctl is more than a mere command; it is also a kernel call and a kernel interface. Probably the most common use of sysctl is turning on IP forwarding in Linux routers by using the echo command:
# echo 1 > /proc/sys/net/ipv4/ip_forward
Or, you can do this with the sysctl command itself. This example is for Debian:
# sysctl -w net.ipv4.conf.default.forwarding=1
To read current values, use the sysctl command:
$ sysctl net.ipv4.conf.default.forwarding
net.ipv4.conf.default.forwarding = 1
Because the /proc filesystem is a virtual filesystem that does not exist on your hard drive, this setting disappears at reboot. To make it persistent, one way is to make an entry like this in an iptables script:
echo "Enabling forwarding. This box is now a router."
echo 1 > /proc/sys/net/ipv4/ip_forward
That turns on forwarding every time your iptables script runs. To have forwarding on all the time, make an entry in /etc/sysctl.conf:
In /etc/sysctl.conf you may use dots in place of slashes, though either one works fine.
Fedora users must use net.ipv4.ip_forward. How do you know what options to use in /etc/sysctl.conf? That is a Great Mystery, and I fear I cannot point you to a definitive document. However, there is a bit of help here and there. First off, man sysctl shows all eight of the sysctl command options, one of which does not work right. Run this command to see all of your current settings, which will show you the correct options for your system:
$ sysctl -a | less
If you're curious how many there are, run sysctl -a | wc -l. I have 506. You might see some weirdo error messages like this:
error: "Success" reading key "dev.parport.parport0.autoprobe3"
Ignore them, that's just a broken way to say "it worked." "permission denied" errors mean you need root privileges to read those particular keys.
# sysctl -a | grep net.ipv4.conf| less
shows your configurable IPv4 network settings.
The capital -A option is supposed to display the results in a table format, but it doesn't, it's the same as little -a.
sysctl operates on /proc/sys, which you can browse like any other filesystem. Most network and system monitoring tools read from /proc, because that is a live snapshot of your running kernel.
Other than turning routing on or off, what's the point of understanding how to use sysctl? Truthfully, not so much anymore, especially since Linux kernels from 2.4.27 on, and 2.6.7 upwards have TCP autotuning enabled by default. In the olden days admins used sysctl to manually tweak TCP settings to improve performance. TCP autotuning works well and I doubt you'll be able to improve on it. Run this command to see if your system has TCP autotuning enabled:
$ cat /proc/sys/net/ipv4/tcp_moderate_rcvbuf
1 means yes, it is enabled, 0 means no, it is not enabled. To learn about manually tuning TCP parameters, see TCP Tuning and Network Troubleshooting.
The main reason I use sysctl is to tighten up network security. These are entries I usually stick in sysctl.conf on routers and firewalls:
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.ip_dynaddr = 1
Changes to /etc/sysctl.conf are activated immediately with this command:
# sysctl -p /etc/sysctl.conf
Beware of distribution differences- sysctl -a shows you the correct options to use. Want to know what all these options mean? Download the Linux kernel documentation, which on a lot of Linux distributions is a separate package, or download a kernel from Kernel.org and dig it out of the source tarball. Documentation/sysctl/README tells you where to find everything. A good resource is Oskar Andreasson's Ipsysctl-tutorial.