Spam Cleaning with the Big Boys

By Steven J. Vaughan-Nichols | Nov 5, 2003 | Print this Page
http://www.enterprisenetworkingplanet.com/netsysm/article.php/3104641/Spam-Cleaning-with-the-Big-Boys.htm

Your users think they have spam problems in receiving a hundred or so spam messages a day? Tell them to try administering the network for an ISP or business, where on a good day you're fortunate to have to deal with fewer than one hundred thousand spam messages a day.

Can't believe it's really that much? Think again. So just how bad is the problem? Ferris Research, a San Francisco- and London-based email and groupware analysis firm, says that 30% of inbound email is spam at ISPs, while at companies, spam accounts for a full 15% to 20% of inbound email. "In 2002," Ferris reports, "the total cost of spam to corporate organizations in the United States was $8.9 billion."

Since that Ferris study, things have only gotten worse. According to ISP and business mail administrators I've spoken with, ISP inbound mail is now up to 50% junk mail, while corporate e-mail servers are up to a rather horrifying 30%. And with groups like the Australia-based Coalition against Unsolicited Bulk Email estimating that spam's volume is doubling every 4.5 months, the spam problem is only going to get worse.

So what can you do when your network bandwidth is eaten alive by spammers, your users are screaming for relief, and your mail server hard drives are always running close to their limits? What most ISPs and companies are doing is deploying gateway anti-spam programs.

Specifically, ISPs tend to deploy SpamAssassin, a powerful open source mail-filtering program, while businesses tend to favor commercial programs like Brightmail Anti-Spam 5.1 or MailFrontier Anti-Spam Gateway 2.1.

You could, of course, install client-based programs like Norton AntiSpam 2004 or Qurb 2, but that's not a great idea in ISP and corporate environments for a couple of reasons.

The first is simply that client-based approaches cost more – much more – per user than server-based solutions. The other – and really the more important – reason is that supporting them will cost you even more in terms of help desk time.

Thus, while client-based solutions are fine for individual users or even small businesses, they simply don't scale well for ISPs or medium to large businesses.

Page 2: Spam Protection at the Gateway

Spam Protection at the Gateway

Regardless of the specific gateway program used, this kind of software always has several things in common. First, the gateways must run on boxes of their own. You can't run them on the same box as the mail server, regardless of whether you're running sendmail or Exchange 2000.

Next, you should take the memory requirements for any given program and double it on your production machines. It's not that the software won't run properly at the minimal required level of memory; it will. However, these programs need every KB of RAM they can get in order to more quickly weed out the bad mail. These processes take up a lot of RAM, and remember, the spam load is only going to get higher – much higher – in the coming years.

To keep the mail moving in a timely fashion, you'll also need as fast a connection as you can get between your spam killer, Internet gateway, and e-mail server. If you've been thinking about moving to Gigabit Ethernet, well, pulverizing spam is a better reason than most for the upgrade.

You do need to give your users some level of control over the spam process, as one man's spam is another man's steak. There are two basic ways to approach this. One is to simply deliver the spam to the user's client mailbox and set it up so that they can look in their spam mailbox whenever they want.

There are two problems with this approach. The first is that your internal network is still going to be clogged up by spam traffic. The other is that if you're going to go ahead and send all the spam along, perhaps an inexpensive or open source client-based solution like POPFile, would serve your users better.

The second way of handling it is to simply keep the most recent spam mail in a server-based folder for users to look in if they suspect that they've missed a very important message. This option isn't ideal either; in this case, you're committing valuable server disk space to spam.

But it's not as if you have much of a choice. Even with the best Bayesian filters and individually tweaked anti-spam settings, at least one message in a hundred will be misidentified. That's not so bad, especially when it means that your users will see a small fraction of junk e-mail instead of the flood they're used to, but every now and again, a message that needed to get through – a false-positive – will get blocked. It's for those valuable but misidentified messages that you need to give your users some mechanism to look at their spam mail.

Page 3: Set 'em and Forget 'em? No Way

Set 'em and Forget 'em? No Way

You'll also need to plan to constantly look at how well your spam protection is actually working. In my experience, SpamAssassin particularly needs a constant eye on it lest it start letting more spam through while increasing its false positive rate. This is also true of the other programs, but with the commercial products, changes in spam patterns are usually reflected in these programs' regular updates.

The choice is yours. SpamAssassin will run great, especially if you have someone constantly managing it. The others require less attention, but you'll need to purchase a long-term support contract to be safe. To me, the key factor is your network or e-mail administrator's level of expertise. If they're already comfortable working with complex procmail or the like scripting, SpamAssassin is probably the better option. On the other hand, if they're still stumbling around Exchange's graphical interface, it will be more cost effective to go with a commercial program.

Regardless of how you update and manage your spam program, the simple truth is that you simply can't set them up once and forget about them. Just as spammers are always changing the way they send spam, you must constantly be on the alert for these changes and adjust your spam filters accordingly. Yes, it's a pain, but there's no choice in the matter.

Consider two years ago, if an e-mail came in with a valid "From" header, you could safely assume that it was a perfectly fine e-mail. Today, with forged headers being a part of every spammer's toolbox, only a fool would assume that just because the "From" field looks OK that the mail isn't necessarily spam.

You also need to keep your users informed of the ways they can slow down spam. For example, encourage them not to put their real e-mail addresses on public Web sites or postings. Instead, a format like joeREMOVE@vna1.com will let any human reader know that chances are Joe can be reached at joe@vna1.com, while bot programs that collect addresses from the Web will faithfully collect the bogus address.

At the same time, though, some user-based anti-spam ideas actually do more harm than good. For example, sending out fake 'bounce' or 'notice of spam' messages to spammers won't do much good. In the first case, with fake headers being all the rage, sending someone a note falsely telling them that their message didn't arrive or that their message is spam is highly unlikely to actually reach the real sender.

All this will do is eat up more network traffic and annoy the innocent user on the other end of the Internet line. And even if the message does get to a spammer, why in the world do you think they'd care? Spamming relies upon sheer volume; its senders already know that their success rate is going to be in the 0.01% range per message sent.

No, the only real answer is to install a gateway side server program to stop spam and then ensure it's continually managed. You can forget about a magic anti-spam program or law coming along and re-setting the e-mail server clock back to 1997. It's not going to happen, and if you want to keep your users happy and your e-mail costs down, you should put a server-based solution in sooner rather than later.

Feature courtesy of EITPlanet.

» See All Articles by Columnist Steven J. Vaughan-Nichols