Tripwire - The Only Way to Really Know
So you think you may have been hacked, but you're really not sure 'cause some crackers seem pretty stealthy (1). There really is only one way to know - employ a file integrity checker, like Tripwire or AIDE. In this article, I'll explain why you need Tripwire/AIDE, what they do, and how you can deploy Tripwire. I'll give you a sample configuration that you can tune.
By the way, I'd like to thank Ben Tilly, Dr. Ernst-Udo Wallenborn, Matt Gischer, Mike Redan, and Kurt Seifried, who answered my last article's question on the SUID root PAM programs.
Why Do I Need Tripwire?
A cracker breaks into a system by exploiting an already present vulnerability. After he hacks your computer, he'll usually install a rootkit and create or install several Trojan horses. The rootkit replaces many of your system utilities to hide the attacker's activities. For instance, it replaces your ps command with one that will not show the attacker's programs. The Trojan horse (2) programs give the attacker a means to get back into your system with root, so they don't have to use the same exploit over and over. (Sometimes, the cracker will even patch the original vulnerability, to protect his new property!)
|"Often, the cracker doesn't want to disrupt your use/business - he just wants a launching platform for IRC bots, DDoS programs, and sniffers."|
Your first (smaller) problem is this: you may not even know you've been hacked! Often, the cracker doesn't want to disrupt your use/business - he just wants a launching platform for IRC bots, DDoS programs, and sniffers. He'll use his rootkit to stay out of sight and the Trojan horses to regain access to the system without tripping most forms of IDS. But, what if you do manage to realize you've been hacked?
Your second, larger, problem comes in here: you don't know what's changed on your system. Your system diagnostic tools have all been replaced by a rootkit! You can't trust ps, top, w, or even ls...! You need some way of figuring out exactly what files have changed, so you can put things back, patch any vulnerabilities, and trust your own system again. You need a file integrity checker. You absolutely, positively gotta kill every last illegal binary in the room. Accept no substitutes!
What Does Tripwire Do?
Tripwire, the original file integrity checker, monitors changes to the filesystem, including your binaries like ps, w, ls and so on... It accomplishes this by storing extended data on every important file in your system. It stores each file's size, mode, last modification time, and all the other major data in the inode. It also computes a series of "signatures" for each file, which it uses to keep track of changes to the file's contents. You run Tripwire whenever you want to check that no one has tampered with any of these files. Tripwire checks that database to look for changes to file meta-data. It also confirms that file contents haven't changed by re-computing the signature and comparing with its stored value.
These signatures are the output of one-way-hash-functions(3), like md5 and NIST's Secure Hash Algorithm.. Signatures have two essential properties. First, a small change in the file contents produces a substantial change in the signature itself. So, if I make a two character (but dangerous) change to my /etc/passwd file, changing the line.
will change the signature radically. Here are the corresponding md5 values, taken by md5sum:
6e59239d37ebff43e984725530b868a1 /etc/passwd (before) f287eeeb1074bef6e33f075e62bdc6b4 /etc/passwd (after)
The change in the md5 signature is extreme! The other essential property of these signature algorithms is that it is pratically impossible to generate two files with the same signature. If a signature algorithm didn't hold this property, then an attacker could make the two character change above, but then use a program to pad the file with whitespace/noise characters to get the same signature.
Downloading and Compiling TW ASR 1.3.1
We'll now download and compile Tripwire 1.3.1 ASR. I've chosen the Academic Source Release for this article, for simplicity and because it's free for all platforms. I'll cover 2.x, later, as it does include a number of improvements. Everything we do here is extensible to version 2, though. So, first download the Academic Source Release from http://www.tripwiresecurity.com. You should end up with a file, named like: Tripwire-1.3.1-1.tar.gz. Next, untar it in a private directory, like /home/jay:
$ tar -xzf Tripwire-1.3.1-1.tar.gz $ cd tw_ASR_1.3.1_src
Now, make sure to read the README file, which has lots of useful information. Once you're done, we can continue by editing include/config.h:. Change the following lines (lines 106 and 107 in my copy) to use directories within root:
#define CONFIG_PATH "/usr/local/bin/tw" #define DATABASE_PATH "/var/tripwire"
#define CONFIG_PATH "/root/tw" #define DATABASE_PATH "/root/tw/databases"
We'll change the Makefile too before compiling, changing lines 13 and 14:
DESTDIR = /usr/local/bin/tw DATADIR = /var/tripwire
DESTDIR = /root/tw DATADIR = /root/tw
Now, we'll compile and install. Make sure to do the last step as root.
# make # make install # cd /root/tw
We find the Tripwire files in /root/tw, including a great utility, siggen, which can generate signatures for any one file, like this:
# ./siggen -17 tripwire sig1: md5 : 0PzZFU.NRuADv.IIWROXE8 sig7: sha : 0kkeI7Cd9CkQ4vElcgLpcK7SDgA
Write these down! Someone could Trojan the tripwire binary! Let's now write a configuration file, so we can start using Tripwire.
Writing a "Quiet" Configuration File
First, we'll back up Tripwire's original tw.config.
# mv tw.config tw.config.orig
Now, we'll write our own configuration file. Be warned, though! Tripwire is a "noisy" Intrusion Detection System (IDS) when you try to watch the entire drive. "Noisy" IDSs tend to report too many false positives, though most can be tuned to be quieter. Tuning Tripwire mostly involves listing out exception files (or qualities) that tend to change often on your machine. Once you tune Tripwire to your machine, it will be quieter, but this does take work. Let's write the file - pay careful attention to the comments, as they explain rationale.
# "louder" tw.config for SecurityPortal.com article jjb 06/30/2000 # This file is a sample configuration file that you can use for a # stock Red Hat 6.2 system, using Tripwire 1.3.1. # REFERENCE CHART follows - each letter/number corresponds to a # different quality of the file to watch. # # p - Permission and file type (mode) # i - inode number (inode=entry in the filesystem) # n - link count (number of hard links to the file) # u - UID (owner of the file) # g - GID (group owner of the file) # s - size of file # a - access timestamp (last time the file was accessed (RARELY USED!) # m - modification timestamp (last time the file was modified) # c - inode Change timestamp (last time the inode was modified) # 0-9 - "signature" algorithm to use: 1=md5, 2=snefru, 7=SHA-1 # E - Ignore everything # # First, define a number of monitoring levels. # # Essential system binaries should be monitored on all attributes, with a # high level of certainty. We keep only md5 and SHA-1 for now. @@define BIN E+pinugsmc17 # System logs should be allowed to change, and even to switch inode numbers. # The inode modification is because of Red Hat's automatic log cycling. @@define LOG E+pnug # Device files should simply maintain ownership, permissions and such. # It doesn't make sense to monitor contents. We also ignore inode # mod (c) because this changes every reboot. @@define DEV E+pnug # Essential system config files (/etc/fstab, /etc/hosts.allow) should # be watched very closely. @@define CONF E+pinugsmc17 # Most directories need to allow for new files to be added, so we # won't watch size, mod time, changes to the inode, or compute sigs. @@define DIR E+pinug # # Main configuration starts here... # # Monitor the root directory itself, but don't recurse into it. =/ @@DIR # Monitor essential system binaries: libraries and programs. /bin @@BIN /lib @@BIN /sbin @@BIN /usr @@BIN # Monitor the /boot directory, where the kernel et al. is stored. # System.map changes inode and mod time on every reboot, so ignore # these. /boot @@BIN /boot/System.map @@BIN-mc # Monitor /dev, the devices directory. /dev @@DEV # # Granularly, watch the system's config files... /etc @@CONF # mtab holds current mounted volume information. Usually, we should # treat this as a log, since it must change. /etc/mtab @@LOG # passwd and shadow will change on any system with many users, since # you'll be adding users regularly, changing your passwords... If # you're on a system with few users, watch /etc/passwd more closely. # If you don't allow users to change passwords, watch /etc/shadow # more closely. # ( Most people should leave this as is, unfortunately... ) /etc/passwd @@LOG /etc/shadow @@LOG # /home should change often. We can simply watch the directory itself, # but we have to allow for new directories to be created within. =/home @@DIR # lost+found can be watched as a directory with no monitoring of contents =/lost+found @@DIR # mnt contains the system mount points for CD-ROM, floppy,... Barely # watch this... =/mnt @@DIR # The proc filesystem is special and changes a great deal. =/proc @@DIR # /root is root's home directory. We don't generally watch the contents, # but you can choose to do this if you're careful enough when logging in # as root. =/root @@DIR # /tmp should change often and greatly. =/tmp @@DIR # /var is difficult, as it contains logs, mail queues, and mailboxes, to # name a few type of files. # 1) The log directory is hard to watch, because of the log cycling. # 2) The spool/cron should be watched. We can watch very closely if we # are willing to disallow cron. # 3) The spool/mail directory should only be watched for permissions, but # not for content, since mail files will be added with new users. =/var @@DIR /var/log @@LOG =/var/spool @@LOG /var/spool/cron @@LOG /var/spool/mqueue @@LOG /var/spool/mail @@LOG !/var/lock
OK, so we've created the config file. Let's use it!
Initializing the Database
We need to take a "baseline" picture of the filesystem. We'll run Tripwire with our new configuration file in "Initialize" mode, to build up our database.
# cd /root/tw ; ./tripwire --initialize Tripwire(tm) ASR (Academic Source Release) 1.3.1 File Integrity Assessment Software (c) 1992, Purdue Research Foundation, (c) 1997, 1999 Tripwire Security Systems, Inc. All Rights Reserved. Use Restricted to Authorized Licensees. ### Phase 1: Reading configuration file ### Phase 2: Generating file list ### Phase 3: Creating file information database ### ### Warning: Database file placed in ./databases/tw.db_max. ### ### Make sure to move this file and the configuration ### to secure media! ### ### (Tripwire expects to find it in '/root/tw/databases'.)
As suggested, we should move this to secure media. Actually, we should copy the Tripwire binary, the tw.config file and the database to secure media, like a writable CD-ROM, a tape backup, a SCSI drive in read-only mode, or even a read-only NFS mount. In any case, we should also use siggen to store signatures of these three files - write the signatures down, as you want to make sure no one has tampered with your digital versions.
|"Tripwire detects intrusions by watching for changes in the filesystem. It also lets you know exactly what has changed, so you know which programs you can trust once you have been hacked."|
Example: Detecting Intrusion
Tripwire detects intrusions by watching for changes in the filesystem. It also lets you know exactly what has changed, so you know which programs you can trust once you have been hacked. We can see an example of this by making our own (trivial) backdoors. First, let's copy /bin/sh to /sbin/sh, and make it Set-UID root. We'll make a second copy, just in case this is found out, in /sbin/su. Finally, we'll place a backdoor in place of /bin/su, which will now give root access to anyone logging in as "l33t", and will not require a password for the login.
# cp /bin/sh /sbin/su # chmod 04755 /sbin/su # cp /bin/sh /sbin/sh # chmod 04755 /sbin/su # cp -p /root/myrootkit/su-replacement /bin/su
Believe it or not, these are unlikely to be discovered by the system administrator, as simple as this trick (4) is. Now, let's see what happens when we re-run Tripwire in Integrity Checking mode. In this mode, Tripwire simply re-calculates the database and checks it against your stored version for differences. It should find these two pretty easily.
# ./tripwire Tripwire(tm) ASR (Academic Source Release) 1.3.1 File Integrity Assessment Software (c) 1992, Purdue Research Foundation, (c) 1997, 1999 Tripwire Security Systems, Inc. All Rights Reserved. Use Restricted to Authorized Licensees. ### Phase 1: Reading configuration file ### Phase 2: Generating file list ### Phase 3: Creating file information database ### Phase 4: Searching for inconsistencies ### ### Total files scanned: 69689 ### Files added: 2 ### Files deleted: 0 ### Files changed: 0 ### ### Total file violations: 2 ### added: -rwxr-xr-x root 316848 Jul 2 17:11:55 2000 /sbin/sh added: -rwxr-xr-x root 316848 Jul 2 17:12:23 2000 /sbin/su changed: -rwsr-sr-x root 316848 Jul 2 17:47:10 2000 /sbin/su ### Phase 5: Generating observed/expected pairs for changed files ### ### Attr Observed (what it is) Expected (what it should be) ### =========== ============================= ============================= /bin/su st_size: 316848 14188 st_mtime: Sun Jul 2 17:47:10 2000 Sun Jul 2 17:12:23 2000 st_ctime: Sun Jul 2 17:47:22 2000 Sun Jul 2 17:12:23 2000 md5 (sig1): 0nGKgbNQwrqtPUEnjX2YWl 3.w7.UtVCY8J8AyytPc5Gp sha (sig7): 33Vh72XyEIvKGPiOY1Tm8qu.KHU 3MWvwVuy5iRBdHr7Y.lIzdljA1e
Reaction - A Quieter, Quicker Configuration:
The Audit Floppy
Notice that these runs are taking an awful long time. On my Celeron 466 test system, with a full install of Red Hat 6.2 on an IDE drive, each confirmation takes about 10 minutes! Further, the database was about 10 megabytes, which doesn't come close to fitting on a nice, easy to re-write floppy disk! On the other side of the spectrum from a very inclusive configuration, we have the Audit Floppy. We basically watch the system binaries, and the main /lib directory. This configuration is small enough to fit the database on a floppy, along with the tripwire binary and configuration file. It'll also be "quiet," in that it will register very few false positives, while catching rootkit replacements of programs, like ps or w... Since we can set the floppy to be read-only, the Tripwire files are quite safe! The downside is that this minimal configuration will miss changes in configuration files, libraries and other directories.
For now, here's what we do. We'll basically watch the binaries, programs and libraries.
# quieter tw.config for SecurityPortal.com article jjb 06/30/2000 # This file is a sample configuration file that you can use for a stock # Red Hat 6.2 system, using Tripwire 1.3.1. This is a "quiet" version # for an audit floppy. # # First, define a number of monitoring levels. # # Essential system binaries should be monitored on all attributes, with a # high level of certainty. We keep only md5 and SHA-1 for now. @@define BIN E+pinugsmc17 # Monitor essential system binaries: libraries and programs /bin @@BIN /lib @@BIN /sbin @@BIN /usr/bin @@BIN /usr/sbin @@BIN # Monitor the /boot directory, where the kernel et al. is stored. System.map # changes inode and mod time on every reboot, so ignore these. /boot @@BIN /boot/System.map @@BIN-mc
This configuration, by the way, ran on my test machine in 61 seconds! Its database was under 400k, leaving plenty of room on a floppy for the Tripwire binary. There are serious advantages to smaller configurations: for one, they run quickly enough that you can Tripwire once every hour! They are also much, much quieter - if you detect changes with this configuration, you can be fairly certain that you're in trouble5!
Afterword - The Tradeoffs
We had to make serious tradeoffs when crafting the Tripwire configuration file. The more inclusive we make it, the more false positives we'll see; the larger the database gets, the longer the runs will take. What purpose you have for Tripwire influences how you balance this. You might even make two configurations! You could have one config file/database pair that looks like our Audit Floppy, which runs automatically every hour and pages you if a single file changes. Your other config file/database pair could be very inclusive, which you use interactively when you suspect that you've been hacked.
Tripwire's recent Version 2.x mitigates some of these tradeoffs very well, with some really cool features. One such feature allows you to set priorities and mail alerts on different files, so that you get a page in the middle of the night when an essential binary changes, but only a gentle warning later when your log files get cycled. Their Tripwire HQ product lets you manage all this from a central console, making Tripwire much more scalable - in my opinion, the HQ product is the only good way to deploy this across an enterprise. The downsides to 2.0 are few: 1) Tripwire 2.x costs money for non-Linux machines. 2) You don't get the code anymore, for now, which stops you from making modifications to the program. I'll do a future article on Tripwire 2.x, though, so stay tuned. The folks over at Tripwire are doing some very interesting work!
Jay Beale is the Lead Developer of the Bastille Linux Project (http://www.bastille-linux.org). He is the author of several articles on Unix/Linux security, along with the upcoming book "Securing Linux the Bastille Way," to be published by Addison Wesley. At his day job, Jay is a security admin working on Solaris and Linux boxes. You can learn more about his articles, talks and favorite security links via http://www.bastille- linux.org/jay.
1 Actually, most of them aren't stealthy. They just have nice tools, like log-wipers and rootkits!
3 For the mathematically inclined, one-way-hash-functions are generally thought to have few to no collisions, such that each hash value (signature) computed corresponds to about 1 possible file content set. (nearly 1-to-1)
4 su won't change behavior, because the real su is first in root's $PATH variable.
5 Unless you just patched your system...make sure that you run Tripwire before you patch, then re- initialize the database immediately after.
SecurityPortal is the world's foremost on-line resource and services provider for companies and individuals concerned about protecting their information systems and networks.
The Focal Point for Security on the Net (tm)