Lax SSH Key Management Endangers Enterprise Networks
Eighteen years ago, Tatu Ylönen invented the original Secure Shell (SSH) protocol in response to a password-sniffing attack at the Helsinki University of Technology. Since then, the technology has become ubiquitous. But Ylönen, founder and CEO of SSH Communications Security, now warns that poor SSH key management is a "ticking time bomb" within the enterprise. What's worse, it's a ticking time bomb that regulators like the PCI Security Standards Council haven't quite committed to defusing.
SSH and the automated network
The risks, Ylönen stressed, arise not from flaws in the SSH protocols, but from what he called "negligence in how people manage access for automation."
SSH is a network-level encryption protocol that resides largely beneath the end user's line of sight, in the administrator domain. The protocol has a variety of applications, such as file transfer, backup, system administration, and network administration. Among those applications, SSH's widespread use in router, virtualization, and cloud services management—in particular the provisioning of new virtual machines within cloud infrastructures—makes it increasingly important to the modern enterprise. Machine-to-machine connections now dominate network environments. Those machine-to-machine connections require vast amounts of automation. And therein, Ylönen told me, lies the rub.
"If you automate something in a data center, you have one computer basically logging into another and then doing something to it. This login requires some kind of authorization, usually using SSH. In most organizations, the way those authorizations are configured hasn't been managed at all. Those credentials are a ticking time bomb," Ylönen said.
The risks of unmanaged SSH keys
The dangers posed by poorly managed automated access credentials originate in large part from their vast numbers. Automated SSH access keys, Ylönen told me, typically far outnumber those assigned to individuals within any given organization.
To demonstrate this, Ylönen described a real-world scenario faced by an SSH customer. This customer, a major financial institution, has about 100,000 employees, but over 1 million SSH keys granting access to the institution's production servers. "We're talking about 10 times more key-based access credentials than they have passwords in the organization, and when we counted logins in their environment, including system-level logins into the production servers, we found that 80 to 90 percent of those were automatic," he said.
"You cannot ignore 90 percent of your credentials or 80 percent of your logins and pretend that they don't happen" just because they are automated, he said. Especially not when the keys in question unlock an enterprise's most mission-critical infrastructure.
"Time after time, we're seeing situations where logging in with automated SSH credentials from test systems or development systems allowed us into production, using keys with no additional user authentication. You can break into a test system and get into the production environment from there," Ylönen said. In many cases, the credentials allow this by providing access to one system that possesses automated credentials to then grant access to others, which then open doors to yet more machines or group of machines, and so on, so that an intruder can penetrate everything within minutes.
"You could reconfigure the network, the routers, the virtualization platforms. You could compromise the servers that hold critical data. You could get into an Oracle database on the database level, read everything, modify anything, falsify anything, destroy anything," Ylönen said.
Next page: Vulnerabilities already discovered and best practices for mitigating the risks.
Vulnerabilities already discovered: F5 Networks and the U.S. emergency broadcast system
The risks Ylönen discussed aren't theoretical. SSH vulnerabilities have already come to light. In June of 2012, an SSH vulnerability was discovered in a number of F5 Networks platforms, among them fifteen BIG-IP models and the BIG-IP Virtual Edition. The vulnerability arose from a configuration error that made a private SSH key public; exploiting it would allow remote users to log in with root access on the affected systems. "Neither a strong password nor remote authentication helps mitigate the issue," F5 declared in its security advisory.
More recently, a compromised SSH key appeared in Monroe Electronics products used to broadcast Emergency Alert System messages (EAS). The key granted root access to the devices affected. Exploiting the vulnerability could have compromised the U.S. EAS's "availability, integrity, and confidentiality," according to a Monroe Electronics security advisory.
Best practices for getting a handle on your enterprise's SSH credentials
Alarmingly, most organizations simply don't have control over all those credentials. Most have no idea how many automated keys they have that allow access to their production environments. They don't know who has access to the keys, and they certainly don't know who can access their most business-critical systems and data, according to Ylönen. Large financial institutions and enterprises may have hundreds of thousands of uncontrolled SSH keys on their hands. And, he said, "You can't just remove the keys, because some of them may be critical for running the business."
So what can organizations do to take control of their keys?
SSH Communications does provide tools to mitigate the risks of automated SSH keys: Universal SSH Key Manager for enterprises, and SSH Risk Assessor for security and compliance auditors. He stressed that organizations will still have "a lot of manual work to do," however.
Organizations must first get the creation of new SSH keys under control with a standardized provisioning process. "You wouldn't give people user accounts on your servers without filling in certain paperwork and providing proper reasons for granting access. The same needs to apply for key-based access," he said.
Next, Ylönen advised a full examination of all an organization's keys to understand which of the keys are actually in use. With that understanding, administrators can then delete the keys that are no longer needed and "justify, document, and audit" the remaining keys, he said. Documentation and auditing are particularly vital "so that if a hacker installs a new key as a back door, we immediately detect it," Ylönen explained.
From there, "organizations need to configure restrictions on what can be done with their keys," he said. Keys often provide unlimited privileges and can get command line access to critical systems. To mitigate the risks that that incurs, Ylönen advised that "access needs to be restricted so that a given key can only run a single, pre-specified command within a machine, so that you cannot use it to spread an attack."
Next page: What regulators are doing, and the role of the PCI Council
What regulators are doing, and the role of the PCI Council
All those steps require a significant investment in money and manpower, of course, particularly for a risk that can be "hard to quantify," Ylönen said. That's why many organizations "don't do anything as long as they can get away with not doing anything," he added. And they'll continue to get away with not doing anything for as long as they can.
For the past year, Ylönen has been on a crusade to change that. He's been traveling and communicating the risks to various standards organizations. The U.S. government, he said, is taking the matter seriously: within the next few weeks, the National Institute of Standards and Technology (NIST) will release a document with guidelines on how to manage automated access via SSH. In addition, the Internet Engineering Task Force (IETF) has released a draft of SSH key management guidelines, which Ylönen co-authored.
The Payment Card Industry council, however, has lagged behind, according to Ylönen. While the PCI-DSS requirements are generally good, he said, "the PCI is being almost negligent in addressing automated access more properly in the standards."
The current standards focus primarily on access to systems by people rather than access by machine, since at the time the current standards were drafted, virtualization and machine-to-machine access weren't as prevalent as they are now. Compounding the problem are the people in charge of the PCI standards. They come mostly from a background of payment terminals and Windows-based systems, Ylönen explained. SSH applies more to infrastructure, virtualization platforms, and Unix or Linux environments, requiring a different kind of expertise. And the security issues have so far failed to catch enough PCI interest to drive a change to the standards.
Additionally, lack of knowledge or exposure at the auditor level hinders enforcement of what relevant standards the PCI-DSS does currently have. Some PCI auditors do understand automated access and SSH keys, he conceded. Most, however, don't. "If the PCI standard says you have to manage authentication credentials, then they think passwords, security tokens, and so on. They don't think of SSH keys, so they never actually know to ask about SSH keys during an audit. They don't address automated access and credentials, even though those have become the majority of access in many environments in the last five years," Ylönen said.
Ylönen expressed frustration at this. This week, he attended a meeting of the PCI Council in Las Vegas, where he hopes to convince regulators to strengthen PCI-DSS rules around the management of SSH credentials. The trip coincides with the announcement of SSH Communications Security's new Auditor and Compliance Education Program, aimed at helping implement new regulatory requirements to strengthen access control and the security of encrypted networks.
"I'm worried that if nothing is done about this, we'll see major incidents within the next five years, the lifetime of the current round of PCI standards," he said. The next round of PCI standards, PCI-DSS 3.0, will go into effect in mid-2015; Ylönen later clarified that "if the SSH key issue does not get sufficiently addressed on this round (3.0), the next PCI-DSS version will likely be three years from now and come into force mid-2018." He added that "I don't think it can wait that long."
Editor's Note: This article was updated to reflect a clarification provided on the timeframe until the next round of PCI standards mentioned in the last paragraph.
Jude Chao is executive editor of Enterprise Networking Planet. Follow her on Twitter @judechao.