Scripting Clinic: Nagging Logs Make for Safe Networks

By Carla Schroder | Oct 20, 2004 | Print this Page
http://www.enterprisenetworkingplanet.com/netsysm/article.php/3424451/Scripting-Clinic-Nagging-Logs-Make-for-Safe-Networks.htm

It is delightfully easy to ignore log files. In fact that is the easiest part of the hardworking network or system administrator's job. But of course it is a law of nature that such ease must be punished in the form of unexpected crises, and there we are standing in the rubble weeping "If only I had paid attention to my logfiles, I wouldn't be in this mess."

When you want to keep an eye on multiple logfiles, keeping a bunch of terminal windows open can become impractical. An alternative is to send the output of several logs to a single console or terminal.
To avoid this dreadful scenario you have a number of options. One is to change to a different career, one that does not involve computers. Another option is to train your logfiles to demand your attention. Let's take a closer look at the second option.

Tailing Logs

A simple way to keep an eye on logfiles in real-time is with the tail utility. tail displays the last ten lines of a file. The -f flag (follow) updates the display as the file changes:

# tail -f /var/log/auth.log
Oct 18 19:25:52 windbag su[5420]: (pam_unix) session opened for user root by (uid=1000)
Oct 18 19:26:38 windbag su[5431]: + pts/3 root:testfoo
Oct 18 19:26:38 windbag su[5431]: (pam_unix) session opened for user testfoo by (uid=0)
Oct 18 19:33:29 windbag su[5438]: (pam_unix) authentication failure; logname= uid=1000 euid=0 tty=pts/3 ruser=carla rhost= user=root
Oct 18 19:33:31 windbag su[5438]: pam_authenticate: Authentication failure

There is one more option you should add to this. tail -f follows the file descriptor. File descriptors are not the same as filenames. The filename can change, but the file descriptor remains the same. Since most logs are configured to rotate, eventually tail will be staring blankly at an archived, inactive log. To make tail follow the live logfile, use the name keyword:

# tail --follow=name /var/log/auth.log

This makes tail periodically close and re-open the file, so it will always display the current working logfile, even after the log has been rotated and archived.

Tailing Multiple Logfiles

When you want to keep an eye on multiple logfiles, keeping a bunch of terminal windows open can become impractical. An alternative is to send the output of several logs to a single console or terminal. To do this, open /etc/syslog.conf, and configure the logs you want to monitor to also send their output to your console or terminal.

*.*;auth,authpriv.*         -/var/log/syslog
*.*;auth,authpriv.*         /dev/pts/7
kern.*          -/var/log/kern.log
kern.*          /dev/pts/7

Restart syslogd:

# kill -SIGHUP `cat /var/run/syslogd.pid`

This is what you see on /dev/pts/7:

root@windbag:~# Oct 18 20:36:28 windbag kernel: via_audio: ignoring drain playback error -11
Oct 18 20:43:40 windbag syslogd 1.4.1#15: restart.
Oct 18 20:44:00 windbag su[6480]: + pts/1 carla:testfoo
Oct 18 20:44:00 windbag su[6480]: (pam_unix) session opened for user testfoo by (uid=1000)
Oct 18 20:44:41 windbag kernel: via_audio: ignoring drain playback error -11

How do you know what /dev designation to use? Easy as pie: from the console or X terminal you want the log messages to appear in, run tty:

root@windbag:~# tty
/dev/pts/7

Remote Logging

Sysadmins must be free. You can send log output all over the place just by editing /etc/syslog.conf. Let's send kernel messages to remote host frodo, and don't blame me for the recent population boom of "Lord of the Rings" fanatics. Who probably haven't bothered to read the books anyway. Hmph.

Edit /etc/syslog.conf to tell it where to send log outputs:

kern.*          @frodo.shire.net

Restart syslogd:

# kill -SIGHUP `cat /var/run/syslogd.pid`

Then on frodo, you must run syslogd with the -r flag, to enable it to receive from remote hosts. Kill it, then start it up again:

# kill `cat /var/run/syslogd.pid`
# syslogd -r

/etc/syslog.conf lets you configure eight different logging levels: debug, info, notice, warning, err, crit, alert, and emerg. For more information on tweaking syslog.conf, read the syslog.conf(5) man page (man 5 syslog.conf).

Continued on page 2: Email Notifications With Logwatch

Continued From Page 1

Email Notifications With Logwatch

Logwatch is a slick Perl script that bundles up logfile reports and emails them to you. Debian users can install it by running apt-get install logwatch. Debian puts the configuration files in /etc/logwatch. The RPM puts them in /etc/log.d. Of course you may also install from sources. Be sure to consult the README for installation.

To make it go, first find logwatch.conf. You'll need to make a few tweaks. Set the "MailTo" directive to your desired email address, or local account. For local mail, most Linux systems still come with venerable old "mail", which works just fine:

MailTo = carla
mailer = /usr/bin/mail

Of course you may use any mailer you wish.

To make Logwatch send you daily reports, set the time range to "Today":

Range = Today

Other choices are "All" and "Yesterday." Now set your desired detail level for your reports:

Detail = High

Save your changes, and run Logwatch to send you a report:

# logwatch

The whole idea is to have Logwatch work without you having to exert yourself, so now you have to edit /etc/crontab to run Logwatch at your desired intervals. This runs it daily at 1am:

# m h dom mon dow user	command
   0 1	* * *	root       /usr/sbin/logwatch

Logging Strategy

There are a lot of different ways to tweak log output. Logwatch and syslog both have a large number of configurable options. I like to configure syslog.conf for more detailed output, then trim it back in Logwatch. That way I get a nice summary from Logwatch, and if there is anything scary that needs investigation, the regular system logs will tell all.

Resources

  • See the man pages for tail, syslog, and syslog.conf.
  • Logwatch resides at Logwatch.org.
  • See the man page for mail. If you have mailx on your system, look for /usr/share/doc/mailx.
  • Linux in a Nutshell, by Ellen Siever, is my #1 indispensible Linux command reference