Going About Patch Management the Right Way

White Mechdyne logo in black square

IT support personnel know that when their network is vulnerable, the entire organization is vulnerable. In order to keep systems operating smoothly and ensure network stability and security, organizations need to establish thoroughly documented patch management procedures before applying security or software changes to their IT infrastructure. An effective strategy is crucial especially for organizations with networks hosted at various sites, running various domains, protected by various firewalls and delivering access to geographically dispersed offices as well as home and mobile users.

Unfortunately, patch management is not simply the practice of running a scan and applying whatever new fixes are identified. On the contrary, it is highly recommended that patches be installed in a test environment where they can be verified for compatibility, undergo an approval process or be deployed in multiple steps. That’s why network engineers typically conduct testing using patch management tools like SCCM for Windows patches or third-party alternatives such as Kaseya, SolarWinds, and LANDESK. Failure to perform patch testing before deploying software patches can have companywide implications that interrupt network performance and access for the end user community and lead to volume spikes in contacts at the service desk as well as increased operational costs. Deskside Specialist Rico Feliciano elaborates from first-hand experience. “Even with a Microsoft patch, you can’t skip that testing and validation step,” warns Feliciano. “One client unceremoniously pushed a patch to their own network and it wreaked havoc. The solution was a quick uninstall of the patch, but while we were identifying the root cause, many end users were thoroughly inconvenienced.”

So prior to applying a patch, the infrastructure services team takes a system snapshot, applies the update, and performs application usage testing and validation to ensure all features are functioning correctly. In the event that a system update is unsuccessful, the team can restore the system (snapshot) to its original state prior to commencing the update. Below are the essential patch management procedural steps designed to minimize risk and ensure a smooth patch deployment.

  1. The network support team will download and stage the monthly updates that are released by the software provider (i.e. Microsoft) on a specified day for any given month such as the second Tuesday of the month.
  2. The new updates are then first deployed to a single workstation for the initial validation cycle in order to conduct basic network connectivity testing post update installation. Once the validation cycle is completed and no negative impact is observed, the network support team will update the incident appropriately and request permission for pre-pilot testing.
  3. The pre-pilot validation cycle includes several devices for an advanced level of validation which will be performed by client staff using internal applications. Once this testing cycle has achieved 90% deployment status and has passed with no negative impact, the network support team, often acting under the approval of the CTO, will proceed to full pilot validation cycle. If, on the other hand, the client still maintains legacy assets that are no longer being supported (Microsoft XP, for example) and are no longer compatible with the latest updates, the team should make recommendations for a technology refresh.
  4. The pilot validation cycle can include a specified validation team or group of users to further test the impact of the new updates. Once this testing cycle has achieved 90% deployment status and has passed with no negative impact, the team will proceed to the full deployment cycle provided Enlivant has granted approval documented in the incident.
  5. The full deployment cycle is defined as the newly staged updates being released to all desktop and laptops devices throughout the enterprise which are managed by the ABS solution.
  6. The network support team also notifies clients of any out-of-band (outside of the established schedule) emergency patches. Once securing client approval, the team deploys these emergency patches as required as opposed to waiting until the next patch cycle.

Ultimately, a successful patch management process is the one that mitigates risk to all IT services, functionality, and data security with backout contingencies for even when those best-laid plans go awry. Since the new technology being introduced to the network may have the occasional unforeseen impact, it’s up to the people who consistently follow the above procedural steps to ensure things run smoothly.