Patchy Patching: IT Pet Peeves, Part 4
Poor patching practices top our list of IT pet peeves, and for good reason. They lead to massive security risks.
This month, Enterprise Networking Planet has been counting down its Top Ten IT Pet Peeves. If you missed them, feel free to go back and check out Parts One, Two, and Three.
And now, we present the moment you've all been waiting for – our Number One IT Pet Peeve.
Drum roll, please.
1. Patchy Patching.
"Simon," who has asked that his identity be kept concealed, has decades of experience in IT. As readers of previous installments may recall, he currently works as a security threat analyst for a major Fortune 50 tech company.
Simon has sounded off on other pet peeves for our series, but when I ask him what problem he sees over and over again that ticks him off the most, Simon can only think of one thing: patches.
Lamenting the slow speed at which security patches tend to be applied in many organizations, Simon relates, "Far too often, I see attacks carried out against vulnerabilities that have already been patched."
Indeed, this makes sense if you're a hacker. If you didn't discover a security vulnerability on your own, you can easily find out about it when a vendor releases a security update. At that point, for hackers, it's a race against time as they seek out victims who are slow to patch.
Not to pick on a single vendor, but many, for instance, are the victims who fall prey to vulnerabilities in Adobe software alone, vulnerabilities that had already had patches released for them, but which the victims neglected to patch in time (in the case of at least one company, despite paying a security consulting firm several thousand dollars to review its security practices). This is, depending upon your point of view, either inexcusable or perversely inevitable, considering Adobe's notoriety for relentlessly nagging reminding users to update and patch their Adobe software.
It is not enough, however, to update your systems and move on. Patches and updates also require testing. We need look no further than recent headlines to understand why. Earlier this summer, Verizon's billing system suffered a major, brand-damaging outage as a result of the company failing to properly test an update to its enterprise software.
Simon points out that this is an inevitability in an age when enterprise IT is often vendor-polygamous. Relying on different vendors for different applications and solutions may have its benefits, but one of the drawbacks is that your systems then become subject to more complex requirements to maintain it all.
"If you throw a patched version of one of those applications onto a production server, you risk the possibility that they will no longer play well together – and the possibility of a…crash," says Simon. "An administrator can apply the patches, test interoperability, without risking his company's revenue stream."
A virtualized testbed can go a long way, but Simon argues that good testing processes go beyond this.
"It takes detailed, thorough code reviews, by people with a 'what if' mentality," he says. "One QA lady I knew rolled a Coke can across the keyboard as her first test—if your application failed to recover, you got it back right then."
Simon is not the only IT professional we spoke with whose goat is gotten by improperly maintained software. Steve Athanas, director of systems engineering for the University of Massachusetts Lowell, notes that the esteemed professors with whom he works "get grants all the time that include some stipend or funding for technology."
And what becomes of the (often taxpayer-funded) technology they obtain?
"These systems are almost never backed up, but include some very important research data," says Athanas. "Of course, not only are these boxes not physically safe and secure, but oftentimes, they haven't seen an update cycle or a patch since Bill Clinton was in office."
Athanas cites this as another example of the need to rectify Enterprise Networking Planet's No. 2 Pet Peeve, a lack of user education.
Again, Athanas insists users are not to blame. "You can't blame people in the 1930s for smoking—no one knew how bad it was for you—so until they know better it's not their fault."
Education, of course, requires effective educators, and Simon agrees that IT administrators need to stay educated and up-to-date themselves. Realizing this goal, however, can be daunting.
"[T]here are a number of cases where for various reasons, a system can't be patched immediately," he concedes. "From personal experience, I realize that such an administrator cannot be expected to sift through…thousands of feeds[;] they need a concise list of the current threats, the ramifications of those threats, and how to patch or remediate them."
To this end, Simon recommends that enterprises maintain a patch requirement system that prioritizes various patches and alerts administrators when these patches are released, due, and overdue. These systems, says Simon, can be made remotely accessible while notifying stakeholders and tracking their participation, requiring them to obtain approval for an extension if they were to fail to install a patch within a set period of time. Ultimately, patch requirement systems, according to Simon, leads to decreased periods of time before patches are applied as users, administrators, and other IT higher-ups take more responsibility for security.
Simon provides more detail about patch management systems:
Patch requirements are based on system location. Thus something rated High would only have a limited number of days for a system exposed to the Internet. That same patch, for a test [or] development system, buried deep behind firewalls and router access control lists, might have a much longer time period requirement to be patched. Multiple requirements, based on a system's location between these extremes, [would have] varying time constraints.
Of course, not even these measures are foolproof. Development systems have been successfully attacked before, and routers are notoriously hackable.
The point, however, is not to be foolproof – but to be better. Hence, Simon's advice – once boiled down – is much simpler and to the point.
"Patch!" Simon exclaims. "Patch regularly! Patch quickly! Don't wait! Stay up to date! That is the best advice [I] can give…!"
Photo courtesy of Shutterstock.
Joe Stanganelli is a writer, attorney, and communications consultant. He is also principal and founding attorney of Beacon Hill Law in Boston. Follow him on Twitter at @JoeStanganelli.