If you’re responsible for a corporate network then you’ll be no stranger to logs: records of events that occur on your network, generated by anti-virus and other security software; devices like firewalls, intrusion detection systems (IDS), routers and other networking equipment; server and workstation operating systems; and applications running on your network.
In a large network the number of logs generated every second can run in to the hundreds of thousands, which begs a few important questions: Should you be closely monitoring these logs, and if so, why? And, perhaps most importantly, how?
The question of how logs can be managed is an important one because if it was easy then it already would be as a matter of course. But the fact is log management is far from straightforward. Ultimately, it comes down to finding a way to wade through a continuous stream of logs generated by different systems and spot the ones that are important, using limited log management resources. This is made more difficult by a number of factors, including:
- Large numbers of log sources;
- Inconsistent log content generated by different devices;
- Different log formats;
- Inconsistent time stamps on logs;
- Huge volumes of log data; and
- The need to maintain the confidentiality and integrity of logs.
Luckily, the “how” question is the easiest one to answer. There is no shortage of companies offering log management solutions. What these products do is centralize the logs by collecting them from the many different log sources on your network, normalize the logs so that they are consistent in terms of format and timestamp, and then convert the log data into decision support information: dashboards, charts and so on, to make the log information comprehensible and actionable.
It’s also possible to cook up your own log management system, but it’s likely to be far more efficient and cost effective to buy one off the shelf or to make use of one operating in the cloud either monitored by a managed service provider, or by your own IT staff.
The ‘why’ of log management
This brings us to the question of why logs should be monitored in the first place. The obvious answer is for security purposes; to spot suspicious events such as repeated failed login attempts or port scans. But Mandeep Khera, chief marketing officer at LogLogic, said that, for many companies, (especially SMBs) compliance is the key driver for adopting a log management product.
That’s because they may have no choice but to store and analyze certain logs in order to comply with Federal legislation and regulations including the Federal Information Security Management Act of 2002 (FISMA), the Health Insurance Portability and Accountability Act of 1996 (HIPAA), the Sarbanes-Oxley Act of 2002 (SOX), the Gramm-Leach-Bliley Act (GLBA), and the Payment Card Industry Data Security Standard (PCI DSS).
Of course, some compliance regulations — PCI DSS is a good example — are all about security anyway. But would companies use log management products if it wasn’t for compliance regulations? “Bigger companies, yes, but for the smaller companies, probably not,” Khera admits.
For these larger companies, Khera said log management systems offer significant security benefits including the ability to spot advanced persistent threat (APT) type attacks, which it may not be possible to detect any other way.
“There will be signs in the logs, but you need to be able to get your data into a form that makes intelligent decision making possible, or that will raise an alarm automatically,” he said. Systems like LogLogic’s are able to accept 150,000 logs per second, search 100 million logs per second, and handle networks generating 60 billion logs per day.
But John Kindervag, a principal analyst at Forrester, said that most companies are misled into thinking that a log management system will help them with their security efforts.
“Vendors talk up the value of a log management system for threat protection or incidence response, but the real value is not that at all,” he said. “The main reason to buy one is simply for compliance reporting; to provide compliance reports to auditors. “
The second reason is one of internal politics. Specifically, to help security teams justify their own value, he said.
“These teams need to be able to stop having their budgets cut on the grounds that there haven’t been any security problems. Log management allows them to show data that proves the number of times that an organization has been attacked,” said Kindervag.
In terms of actual security, Kindervag believes that systems such as firewalls, anomaly detection systems, IDSs and Web security appliances are much more valuable, because they can stop malicious attacks before they can cause damage.
“You have to be able to stop the bad guys on the wire. If you are logging an attack then it is already too late” Kindervag said. And he dismisses the idea that log management systems are the best or only defense against APTs. “Any Layer 7 device, like an IPS [intrusion prevention system] can detect command and control traffic. Proactive controls are much more important.”
But this argument is contested by Ross Brewer, a vice president at LogRhythm.
“Big breaches have led many organizations to recognize that firewalls, IDS and other single point perimeter defenses are failing, and failing massively. They now need a way to respond to attacks automatically,” he said.
As well as carrying out forensics by building a picture of how an attack was perpetrated and what damage may have been done by analyzing logs, log management systems help detect many sorts of ongoing threats, from APTs to rogue administrators. When they do, they can respond with alerts or direct actions.
“If I appear to log in to the network from Hong Kong, and then 10 minutes later I log in from London, the system will automatically spot this. It can then automatically disable the account, or send out an alert,” he said.
In fact, LogRhythm comes with about a hundred “generally discernible patterns” such as port scans that it is programmed to recognize from logs, and react to. As a result, Brewer believes that systems like LogRhythm should be a key part of an organization’s defenses.
“At the turn of the millennium, log management was about security. Then, from about 2002, the market shifted from security to compliance. But now there’s been another fundamental shift and log management is moving back into the realm of cyber-security.”
Vendors and pricing
Log management systems are available as software, virtual appliances, or physical appliances that sit on your network, and which can be connected together to provide scalability. Increasingly they are being provided in the cloud, as well.
Pricing may be done in a variety of ways including per appliance, per log source or by the volume of logs collected per day. As a rough guide, a system for a company with a few hundred employees may cost around $30,000 to $70,000. A large company with 10,000 to 50,000 employees may be looking at $500,000, while for a very large enterprise a log management system is likely to cost in excess of $1 million.
Key log management vendors include:
Nitro Security (part of McAfee)
NetWitness (part of RSA)
Paul Rubens has been covering IT security for over 20 years. In that time he has written for leading UK and international publications including The Economist, The Times, Financial Times, the BBC, Computing and ServerWatch.