Daemons Running Amok? Daemontools to the Rescue!
If you enjoy futzing with rc files, and inittab, and init.d, and ttys, this may not be your cup of tea. If you enjoy having a bit of free time, and actually feeling caught up on occasion, read on. Last week we looked at tcpserver, that nifty utility for managing TCP services. This week we'll dig into DJ Bernstein's software store for a central daemon administration tool: daemontools.
daemontools makes the overworked, harassed sysadmin's life a whole lot easier. It is a collection of small, specialized utilities that monitor and manage all services from a single directory, /service.
-svscan starts and monitors services. It finds and starts new services right after they are added
-multilog saves error messages to one file or many files. Logging directly to file is faster than logging through syslog. multilog rotates logs to minimize disk usage; if a disk fills up, it gracefully retries, without goofing up the system or losing the data
-supervise handles starts and restarts. Its most charming feature is automatically restarting a service when it dies unexpectedly
-svstat reports on the status of services managed by supervise
-svc adds command-line tools for services monitored by supervise. svc provides a lovely, direct, simple command syntax for stopping, starting, and restarting services
Get the latest version, currently 0.76. Anyone running an older version should consider upgrading, especially .70 or older, as it has been considerably improved.
As always, follow the installation instructions carefully, or it won't work. svscan will start automatically after installation. The installation program edits /etc/inittab to start svscanboot at boot, which then starts up svscan. Check after installation:
$ ps ax | grep svscan
832 ? S 0:00 svscan /service
Installation is usually the best-documented part of djbware, after that it gets rather terse. We'll cover operations in more detail.
See djb's Web page describing his own Unix root filesystem structure. Bookmark this page (see Resources) for reference, it's needed when using djbware to avoid name confusions and collisions. The directories you'll encounter most often are:
As anyone who frequents Unix forums knows, wars rage over directory structures and names. Some don't care for djb's strict requirements, and want the freedom to put things wherever they want. (OK, so go move /bin/login...) The advantage of the djb way is simplified cross-platform services administration- a consistent administrative interface across all unixes. daemontools is strict on where it will install, so it will be the same on every system- BSD, Linux, Solaris, and the many permutations and variations therein. Otherwise, organize your systems any way you like. It doesn't matter to daemontools, as it is mainly a collection of symlinks.
You'll recall the first installation chore was to create the /package directory. /package provides a global namespace for installed djbware, and any other apps that conform to djb's package scheme. All of djb's software knows how to put itself in the correct places in the /package directory hierarchy. By design, the directory structure itself becomes the package database. Use standard filesystem utilities to find and manage things. The daemontools installation creates the /command and /service directories. /command contains symlinks to the binaries in /package.
The /service directory is the heart of daemontools. svscan looks here for services to start. /service is a collection of symlinks to service directories, created by the admin. It doesn't change the existing filesystem structure, it provides a central point of administration. The symlinks must point to a directory. This is the most common mistake for newbies to daemontools- don't link to a file.
For example, I created /etc/sshd as my ssh service directory. It contains the run script, /etc/sshd/sshdrun. Don't forget to make your start scripts executable:
chmod +x sshdrun
Create a symlink to /service:
ln -s /etc/sshd /service/sshd
And that's it- within five seconds or so, svscan will find it and start it up. To verify, use svstat:
# svstat /service/sshd
/service/sshd: up (pid 17387) 58 seconds
daemontools will add a /supervise directory to /etc/sshd:
/supervise contains four files:
control lock ok status
Don't touch. There's nothing to do with these, except verify they exist.
Now here's the part I really like: stopping services without having to grope first for the PID:
#svc -d /service/sshd
This stops it permanently, it will not restart until you tell it to:
#svc -u /service/sshd
Quick and easy. Processes restart in a clean state, just like after booting.
svscan skips hidden directories (dotted directory names.)
Removing a service is easy:
#svc -d /service/sshd
And that's it. Nice and clean, no messing with system files or scripts.
Create a log directory, like /etc/sshd/log. Create a run script, /etc/sshd/logrun. A simple multilog script looks like this:
exec multilog t s1000 ./main
t = timestamp, s1000 limits the size to 1000 bytes, and the dot means automatically create a log rotation in the /main directory.
There also needs to be a command in the run script of the daemon ( /etc/sshd/sshdrun) telling it to generate a log. This varies by program, usually there is a switch that sends logging to stderr instead of syslog.
Backgrounds and Foregrounds
One of the core concepts of daemontools is no background processes. All processes must run in the foreground, so that supervise can monitor them. From the daemontools FAQ:
"How can I supervise a daemon that puts itself into the background? When I run inetd, my shell script exits immediately, so supervise keeps trying to restart it.
"Answer: The best answer is to fix the daemon. Having every daemon put itself into the background is bad software design. "
There is a utility included for making background processes work under daemontools, called fghack. It forces them to run in the foreground, however, supervise cannot send signals to processes running under fghack. Many services, for example BIND, have command options to run either in the foreground or background.
The trickiest part of managing services is writing and editing the run scripts. Most programs come with perfectly good scripts. After that, it's as simple as creating a few symlinks. Potential gotchas: watch out for services that need hard links instead of soft links, such as running inside a chroot jail. File ownership and permissions can get tricky. Pay careful attention to these details, and all will run well.