In-Depth
The 10 Essential Rules of Patch Management
The more work you put in up front, the easier it will be to get and stay current when the patches hit.
It is the lament of the harried Windows IT ace: So little time, so many patches. And it's not getting better any time soon. The sorry fact is, patches are coming in a steady flood, they must be deployed more quickly and few seem to be throwing enough resources at this ever-worsening problem.
The root of the problem is the very patch cycle itself, otherwise known as Hole, Patch, Exploit—Repeat!
Here's how it works. In the very best case, a reputable security firm or white hat hacker discovers a vulnerability and alerts the vendor who can build a patch. The public only finds out about the problem when the patch is released—exactly when the bad guys catch wind.
Meanwhile, hackers dissect the patch to find the flaw, then write code to exploit it. "Sometimes I wonder if, by Microsoft pointing out that there's a hole, some 18-year-old hacker in a black-light illuminated basement says, ‘Aha! Now that I know about the hole I've got those *#*@(#)!'" says Bill Heldman, a senior IT technologist and Microsoft MVP.
Hackers know that on-the-ball shops will wait for conflict testing to install a patch and, even then, may miss a machine or two. If the hack comes before the patch is installed, IT loses.
Even more so, hackers prey on those too lazy, blasé or ignorant to patch. Here the patch ultimately creates a large, gaping and sometimes lasting opening into these networks.
This cycle goes on and on and on and, along the way, hackers find ways to exploit holes more quickly.
It's gotten so bad that there are even patches that patch patches!
Some simply react to this cycle, patching after the whole world has been hit. That's a bad approach—you want to try and stay ahead of the curve.
Compliance regulations such as Sarbanes-Oxley and HIPAA are also key patching drivers. After feeling the impact of HIPAA, Tim Rice, network systems analyst at the Duke University Medical Center Department of Medicine, says they decided to get serious about patching. Before HIPAA hit, he says "patching was done on a sporadic basis."
"Given HIPAA, that wasn't going to be suitable. HIPAA mandates the network be secure, reliable, and available. Part of that is keeping it patched," Rice continues.
And IT has no time to spare. The longer you wait to patch, the longer you are vulnerable. With automated hacking techniques and fast spreading worms and viruses, there's a darn good chance you'll be hit. You want to get, test and deploy the patch as fast as is reasonably possible.
With that in mind, here are 10 rules of patching you must follow.
1. Be Informed
Everyone knows that patching is important. But not everyone knows how important it is and even fewer are true experts on the issue. You should be. Start with a basic patch education by boning up on white papers, scouring Microsoft.com, and poking around Redmondmag.com and MCPmag.com (click here for loads of essential patch links and info).
Now that you've nailed the basics, it's time to drill down. Effective patching is all in the details. To get ‘em requires lots of research and plenty of sources. Here are some key sources for up-to-date patch data:
- Relationships with vendors, especially with Microsoft, but also with other key suppliers (patching is not just a Redmond issue), security firms and patch vendors.
- E-mail notification; to learn more about Microsoft e-mail notification, and RSS feeds, check out: www.microsoft.com/technet/security/bulletin/notify.mspx.
- Web sites such as NTBugtraq.com, Patchmanagement.org and newsgroups.
- Microsoft Security Bulletins and Service Pack documentation. Every time Microsoft releases a patch, it also posts a Knowledge Base article and releases a Security Bulletin that explains what the exploit is, how it works and what systems it affects (so you know if you really need the patch), how critical it is, and how to verify the patch was properly and completely installed.
2. Determine Whether to Patch
When a hole is found, patching seems obvious, but it's not always so simple. It isn't always best to patch every computer. There's actually an economic calculation that needs to be made. Here, IT needs to look at total hours and money spent patching versus cost of downtime. Here are the key factors to consider:
- Cost of lost data
- Downtime costs
- Cost of repairing problem
- Impact on business reputation
Overall you are looking at the need to have systems available and the actual cost of patching versus the risk of not patching the hole. Does the patch offer a positive ROI?
Besides the basic economics is the need to conduct a basic risk assessment, or deciding whether you really need the patch, and if so, how bad?
"Risk analysis is huge in project management. You first analyze, with the rest of the team, the risks you think are facing you—giving each one a name, an estimated chance of the risk really arising, and a priority," says Heldman. "You then develop either a.) annihilation strategies that keep the risks from arising in the first place, or b.) mitigation strategies that allow you to cope when they do arise."
In most cases, it pays to protect at least the most essential systems. And rather than simply not patching because of the costs and time, IT should look at cost effective patching technologies and highly efficient patch practices.
| A Glossary of Fixes |
Workarounds: A quick fix for a hole; often involves simply shutting off a vulnerable function, such as an entire Web server, which is not always acceptable.
Patch: A fix for a specific hole; Microsoft has levels of severity for vulnerabilities.
Hotfix: Built to fix problems, sometimes it's a fix for specific customers.
Service Pack: Batch of fixes and a few new features released every six to 12 months or so. Besides patches, sometimes these packs include hotfixes and updates. Usually well tested, service packs still require a lot of testing and prep before installation. But waiting that long isn't always wise. Many in IT rely upon services packs for patches.
Update: Fixes for non-security problems.
Critical Update: Fixes for problems deemed “critical” but not security related.
— Doug Barney |
"I had to justify it to my boss. We talked about HIPAA and patching. I did an Excel spreadsheet that looked at 10 minutes of my time per machine and the travel to each machine. Then I looked at cheapest labor for this, and it was still far cheaper to automate. I never had to do a real business case—it just sort of became apparent," says Rice.
3. Survey Your IT Surroundings & Standardize
There's an old rule: You can't patch what you don't know you have. Well, maybe the rule isn't that old, but clearly you can't understand the full scope of the problem unless you know what devices and software are in place. Unfortunately this is a resource-intensive process.
Even more unfortunate—most shops don't have an adequate asset management system. If your system isn't up to snuff, it's time to get back in front of your execs and get them to back an investment in asset management, in addition to patch management.
You need to know what is installed and what data is loaded where.
Asset management is an ongoing process. As systems change—new software installed, patches and service packs deployed—your inventory needs to be in sync.
Once you know what you have, check it all for holes. Like asset discovery, vulnerability assessment is an ongoing process, so figure out how to gather the data, and more important, how to analyze it. And use multiple sources of vulnerability discovery.
Don't wait until there's a problem. Instead, proactively conduct these vulnerability assessments.
You know what you have and what the vulnerabilities are. Now it's time to find out exactly how well your systems are patched.
"If you don't know your environment, you'll have no idea of what to patch or how to patch. For small companies this is a lot easier, but in larger organizations, you need to work in a cross functional team to understand the environment, and recommend and implement proper plans that are appropriate for the work groups you are dealing with," argues Michael Ferguson, Director of Technical Services for Frontier Futures Inc.
A jumble of systems is a mess to patch. It may be too late to standardize what's already in place, but it may be worth the effort. An easier job is standardizing everything that comes in from now on. "We use an imaging system. All machines start from the same base point. We do as little customization as possible. Pretty much everyone gets the same patches," explains Rice.
Make sure your new systems are properly set up and that servers run a carefully designed set of applications, a single app or a single set of functions to support a major app. As much as possible, new systems should all have the same config that is locked down.
Documentation is critical, says Chad Hubbard, a network administrator with Action Performance Companies LLC, a maker of sports memorabilia and figurines. "On occasion you may need to go outside your corporate standards. If a device or software is outside your standard your administrators may not be as familiar with it as with standard equipment. In those rare occasions it's always best to document everything. Document every configuration—hardware, software,everything. This will make everything from patching to everyday administration easier," says Hubbard.
You also want to stabilize your overall network with a solid baseline of patches. "All our machines have the same patches; they behave the same," says Rice.
4. Prioritize Systems
Look around at all your systems. If they are many in number, patching everything at the drop of a hat may not be an option. And because patches can introduce instability, you probably don't want to patch everything all at once.
"There's a difference between immediately applying every patch and applying only those patches that should be installed immediately. Some patches can easily wait—and should—while the IT shops runs through a testing cycle," says Heldman.
You need to set patching priorities. This really goes back to risk analysis. Here, the analysis determines the systems that pose the most risk, including which are easiest to crack and which contain the most important information.
The simplest approach is to protect Internet-facing systems that are barraged by hacker exploits. Another key priority is systems that face the public. E-commerce systems that collect dollars, for example, are a no-brainer. But because viruses and hackers can attack anywhere, patching just Internet-facing systems is not nearly enough.
Location is still important, but don't simply protect the systems that seem most geographically at risk. The value of the data is just as important—maybe more so. Simply put, the more important the info, the more protection it should be afforded. Only you and your confidantes can truly figure this all out.
On the other hand, it is okay to pay less attention to non-essential devices and useless data. All data is not created equal.
5. Build a Team & Define Processes
Getting the right people organized the right way is the key to patching efficacy. Here, you survey your organization, rather than your computing assets, to see if you have enough skilled people.
Now you bring them together. One technique is to create a change advisory board or, if there already is one, making patching another of its mandates.
No matter how you structure it, there needs to be clear ownership of the process. You need to have dedicated resources and enough resources to get the job done.
"A patch team is a great idea. Most IT shops have few bodies and little time for a formalized team approach. Nevertheless patch management needs to be something that is defined, formalized and able to be spun out on a moment's notice," says Heldman.
Next, set clear responsibilities. Patch team members may do the actual patching, or in larger shops, there may be a high-level designated group that supports the admins in doing the dirty work of patching.
 |
"We always try to standardize everything, and stick to those standards."
Chad Hubbard
Network Administrator
Action Performance Companies |
"It has to be part of your job. Someone has to have the official assignment to review and evaluate not just the patches but the underlying security considerations and implications that go with them. I have seen too many environments where either everything is patched regardless of the environment, or nothing is patched because of ignorance or laziness," says Ferguson.
Patching processes and procedures should not be crafted or handled in isolation. Patching processes are really an extension of your change management plan. In other words, change management has to adapt to patching. Patching also needs to be written into your security policy and maintenance procedures. If you have the right patching team, this type of integration will come naturally.
Polished processes make patching easier and more likely to be done. "My students understand the importance of patch management but also know that unless it's easy and painless it won't get done in a timely fashion. Good practices fall by the wayside if it seems like a chore," says Carol Miller, a corporate trainer in Canada.
Your internal IT structure will dictate how patching is fundamentally organized. Centralized shops will obviously centralize the analysis of needs and patch testing and rollout. There are advantages to such an approach—patching skills are concentrated and there is a central view of IT assets.
Decentralized shops have a greater challenge. Here, each mini-IT organization has to investigate solutions, survey their inventory and define approaches. Such shops should consider at least an ad hoc patching council that can help standardize technologies and procedures.
Either way, IT has to maintain control, and rule number one here is to never let users do their own patching unless it is under your careful guidance.
| Patch Management
Product Checklist |
IT used to build patch systems using scripts and software distribution systems. Many of these IT pros are now looking at dedicated software or the patch deployment pieces of existing management systems. If you are considering a dedicated solution, here are some things to consider:
- Rollback capabilities— how effectively can damage caused by a patch be undone?
- Security—is the patch server protected; are patches signed for authenticity?
- Scalability.
- Ability to handle low-bandwidth connections.
- How fast does a patch vendor release a patch? (Is there a third-party lag, where the patch vendor evaluates the patch and then releases it to customers?)
- How well are patches packaged for distribution?
- How stable are the released patches?
- Ability to group users and prioritize patches.
- Platform support.
- How good is the library of patches?
— Doug Barney |
6. Automate Via a Good Partner
A good patching process can be labor intensive. Your smartest people spend hours analyzing systems, finding holes, evaluating, testing and installing patches.
None of this increases productivity or provides new features. All it does is keep productivity from waning due to downed systems, and features from failing.
"We still use the sneaker-net plan—patching one system at a time. This is not the best option but seems to be the most reliable for making sure the patch is installed. This type of management is very costly in man hours," says Jim Sanders, a technical administrator with the Oklahoma Housing Finance Agency.
This is where automation comes in. Automated patching software will watch for patches and other updates, categorize them and then analyze your specific vulnerabilities that need repair.
The good ones also provide a simple interface that lets IT view relevant patches and make specific decisions. It should also group computers according to criticality, value of data, exposure, or whatever else you decide.
Microsoft has several automated update services, such as Windows Update Services (WUS) and Software Update Services (SUS), but for some these are simply too basic. "We looked at SUS. The logging was insufficient and there was no granularity of control," says Rice. Third-party patch tools tend to have more functions and tend to embrace far more than just Windows-based systems.
Not everyone has the luxury of a full team, and that's where automation kicks in. "For the department, it is pretty much me. I run the BigFix server for the medical center. Twenty or 30 other people [at Duke] also use BigFix," says Rice. "I am responsible for patching about 1,200 machines."
It takes Rice about two hours a day to handle patching, including reading, browsing Web sites and actually patching. Without automation, it would take "all day long."
 |
"There's a
difference between immediately applying every patch
and applying only
those patches that should be installed immediately."
Bill Heldman
Senior IT Technologist |
"All your systems need to be patched, so get an automated system—it will ease your day immensely," Rice concludes.
Automation is not the sole solution, as you shouldn't trust any third-party tool to solve all your problems. You should decide when a patch is fully tested and safe, when a patch is needed, when a workaround should be used instead.
Total automation keeps IT out of the loop. That's bad because you are far away from the action and not close enough to the nitty-gritty details of vulnerabilities, and can become detached from the danger these openings pose.
Hackers can attack automated patch systems (these systems offer high privileges) so make sure your patch servers are ultra-secure.
Also keep in mind that while vendors test patches, that testing doesn't necessarily match your environment. Testing cannot be fully automated.
The bottom line: Automated doesn't mean automatic. "I am not a fan of automatic systems. You need to test before you apply," says Rice. "Test your patches before you roll them out. That takes the most time. It is surprising where we find problems." During the test phase, Rice will look at workarounds as a stopgap measure.
Manual patching should be used for patch servers and management servers—you don't want them restarted while they are pushing out patches.
7. Deploy (or Not)
All this work and you still haven't patched a darned thing. Welcome to the complex, time-consuming world of patching. But don't fear. You're getting closer.
Before you deploy, you must first decide if you even need to. You must analyze:
- What kind of hole is it?
- How severe?
- How vulnerable are your systems?
- What are the patch requirements?
- What are the dependencies? (What, such as service packs, needs to be in place before the patch can be installed?)
There are three main options:
- Do nothing
- Implement a workaround
- Patch
First, figure out where you are vulnerable. If an exploit targets something you don't use, there is no need to patch.
If you are vulnerable, is it a big deal? If it is, break out emergency procedures.
Sometimes the patch flunks the test. Other times, the patch isn't ready. Because of these lags between exploit and successful patch deployment, IT has to consider other options. Workarounds are the quick and dirty answer.
Workarounds are great, as long as they don't cripple critical systems. For instance, if the fix is to shut down a key port or turn off a feature your users need, then choices have to be made. How serious is the hole, what is the potential damage and how well can your shop operate with reduced functionality?
| Tips & Tricks |
- When a patch is released, examine your entire network for exposure. This should drive your reaction.
- Multi-tiered, in-depth defense can reduce the pressure to rush out the patch.
- Only deploy patches that fit your environment.
- Quarantine assets whose status and configuration is unknown.
- Make sure your systems are secure. A vulnerable system won't be fully protected even with the most up-to-date patches installed.
- Fix the computers that have had a security problem or break-in first.
- Protect patch servers—if these are compromised, everything is vulnerable.
- Like phishing attacks, malware can be made to look like an official patch.
— Doug Barney |
One way to roll out patches is a semi-test, a controlled rollout. This is sort of a rolling test—unfortunately, you still aren't completely patched. This rollout does, however, allow you to take tests along the way. This is fine for low- to medium-level vulnerabilities.
The danger of patch versus danger of exploit is a real Catch-22—unfortunately no one can do these calculations for you.
Lastly, you need a contingency plan for patches gone bad. How can you roll back and how do you protect this now-unpatched computer?
8. Test
The rush to patch can create unstable, damaging code—code that can be more dangerous than the exploit itself. These bad patches wreck otherwise stable systems.
Another issue is that even the best tested patches and service packs were not tested in your environment. The only solution is to test them yourself. "Test your deployment technology, repeat often. Get beta testers that represent every major functional unit," says Ferguson.
He also recommends testing all patches manually. "This may sound a little draconian, but I have seen many deployments fail because the patch was flawed or incompatible with the user's environment," says Ferguson. "Test your deployed patches on your test group. Repeat as needed."
There are many ways to test, and the best one depends upon your environment. Small shops may just roll out the patch to a limited number of clients or servers, which are hopefully not mission-critical boxes. This is the aforementioned controlled rollout. Bigger shops often have separate lab systems that mimic those in production.
 |
"We still use the
sneaker-net plan—patching one system at a time."
Jim Sanders
Technical Administrator
Oklahoma Housing Finance Agency |
Labs often use image-based testing—here, you clone your environment and test against an up-to-date replica. You can clone all your key environments and feed them into lab machines. This lets you test all relevant system variations. Here, tools such as VMware can be useful. Oh, and be sure to keep your clones updated.
"We have 10 instances of VMware workstations and servers. I can take the patches and apply them to the various VM instances to make sure it will not cause the OS any problems. I can't check every app," says Rice.
The degree of testing can also vary. Some test and simply see if the system will reboot. Others run a bunch of tests to see if the patch stands up and the application maintains full function and stability.
Try to isolate all patch code from production while you examine for viruses and malware—and ensure that the patch comes from an authentic source.
After testing, you still need to research what it will take to actually patch these systems. What are the requirements of each patch? Do you have to add a service pack before you can even install a patch?
"Always test before you deploy even if you download the patch from Microsoft. Management won't care that Microsoft said the patch is safe. It's up to you as an administrator to verify that a patch is safe for your systems," says Hubbard.
9. Report and Status Analysis
After a patch is installed, there is still care and maintenance. Here is where acceptance testing and reporting come in. Acceptance testing tells you if the patch was successfully installed, is working, and whether its operation was subsequently disrupted.
Good reporting tools can discern this, but even more so, they can show how many systems got fixed.
Don't stop there. After the patch is installed, continue to check for that vulnerability and reinstall or update the patch if it is disrupted, corrupted, overwritten or otherwise fiddled with.
Also, look for incidents of the patched hole being exploited.
10. Patch and Repeat
After a successful patch deployment, you go right back to the beginning, and:
- Do an inventory
- Do a vulnerability assessment
- Review your organization's ability to effectively patch
And so it goes.
Patching Ain't Perfect
Patch management can't do it all. Patching is just one layer of defense—firewalls, IDS/IPS, education, security audits and anti-virus are still must-haves. On the other hand, simply relying on a firewall, however good, is just as dangerous. And don't forget to have a backup and restore strategy that insures that uncorrupted data is replicated and available for timely restoration.
"Patch management is only one piece of the puzzle for a secure environment," concludes Rodney Landrum, a data analyst and systems engineer for a software development company in Pensacola, Fla. "Anti-virus software and monitoring services provide an overall solution along with patch management."