Improving compliance with faster patch management

Volume and severity of cyber-attacks have increased in the last few years, but a recent report from Ponemon indicates that businesses still maintain their server status quo in terms of software patching and update frequency. Delays in patching, especially public-facing critical servers, leave them out of compliance and vulnerable to the latest exploits.

With current patching strategies, attackers outpace patching frequency and exploit vulnerable servers before administrators can deploy updates. Once an exploit is released, it only takes a matter of minutes for an attacker to scan for vulnerabilities, exploit them, and exfiltrate data, which happens too frequently for current patching strategies to remain effective.

Increase in patching delays and data breaches

Machine learning (ML) and artificial intelligence (AI) aren’t just tools for the good guys. Cyber-criminals also incorporate ML and AI into their own solutions, which makes it more difficult for organizations to stay ahead of the latest vulnerabilities if they don’t have the latest counter defense. According to the latest Ponemon report, 77% of organizations say that they don’t have the resources to keep up with the latest patches before exploits are developed.

In addition to resource limitations, 74% of organizations claim that taking critical systems offline is a primary issue. It’s not only inconvenient but rebooting critical systems is a risk that costs organizations money. There is no guarantee that the system will reboot without any issues, and the longer the system is down, the longer it costs the organization money in downtime. This issue alone is why many administrators choose to delay patching until a reboot can be scheduled and users are aware of the downtime.

The most notable of data breaches due to unpatched software is the 2017 Equifax breach that affected 150 million US, UK, and Canadian consumers. Equifax’s public-facing servers went unpatched for months before being exploited. The cause of the breach was vulnerability CVE-2017-5638, which affected Apache Struts 2 2.3.x before 2.3.32 and 2.5.x before 2.5.10.1. Apache released the security patch in March 2017, but Equifax’s servers had still not been patched in July 2017 when attackers were able to first breach the Equifax system. Equifax monitoring services detected suspicious traffic, but the servers were not taken offline until it was too late and millions of records were stolen.

The WannaCry global ransomware attack exploited unpatched Microsoft machines. WannaCry attackers were able to gain remote code execution on Microsoft machines running Server Message Block 1.0 (SMBv1). Security vulnerability CVE-2017-0143 was published in March 2017 with a software patch, but many global systems were still unpatched in May 2017 when WannaCry was initially deployed. WannaCry rapidly infected hundreds of thousands of machines and caused a global economic loss of $4 billion.

Patching costs and best practices

In the past 5 years, reported vulnerabilities more than doubled from approximately 6500 in the previous few years to over 12,000 in 2019. Because these vulnerabilities are public, it’s critical for compliance and data protection for organizations to patch software as soon as possible. The issue is that installing anything on a production server takes more than a simple command. Administrators must ensure that the patch does not affect the reliability of the system, and downtime should be kept to a minimum.

AT&T and the US Department of Homeland Security published advice for server administrators and several patch management best practices that should be followed. To summarize their advice, organizations should:

• Audit and inventory all digital assets. This step will give administrators an overall outlook on the network’s attack surface and possible risks.
• Assign risk to assets. A risk value will help with patching prioritization.
• Monitor vendors and CVE reports. Vulnerability scanners will find unpatched systems, but administrators should also review the latest reports to find false negatives.
• Test patches in a staging environment. Testing will prevent downtime and reboot failures provided the staging environment is a mirror of production.
• Vulnerabilities with published exploit code should be given priority. Exploit code gives attackers an immediate method to exploit the unpatched system, so the longer a system is unpatched the more likely it will be compromised.
• Use automation to significantly reduce patch time. Automated patching reduces the patching timeframe from months to an average of 48 hours to one week.

Integrating live patching with automation best practices

Automated patch management solves many of the deployment delays seen in the enterprise, but it still does not solve the problem of reboot risks and downtime. Large organizations with hundreds of servers need a solution that doesn’t require much human intervention. Live patching offers a solution to the reboot problem and offers a rebootless patching workflow to avoid downtime.

For many large organizations, rebooting critical servers is out of the question and isn’t an option. Imagine a financial organization such as Visa or Mastercard experiencing downtime due to server reboots. It’s inconceivable that customers cannot use their credit cards due to server reboots. Live patching solves this problem leaving Linux servers up for years without a reboot.

In a use case study, Affinity Water — the largest water supply company in the UK supporting 3.6 million people — uses live patching for their vast network of Linux systems running Red Hat, Oracle, and Ubuntu. Their patching schedule involved a long email trail, documentation, and coordinated efforts across departments. This resulted in long patching delays and left critical billing and consumer servers out of compliance and vulnerable. Now with live patching, Affinity Water no longer schedules reboots, keeps Linux servers patched with the latest updates, and reduced overnight and weekend work for employees.

How does live patching technology work?

In most patching environments, a vulnerability scan finds unpatched servers, the patch manager deploys the patches, and the server reboots if necessary. Live patching software can be deployed using patch management software such as Ansible, Puppet, SaltStack, Chef, or Spacewalk and take over for kernel updates. 

After deployment to servers, live patching modules check for updates at a central distribution server. When a patch is found, the live patching agent:

1. Allocates kernel memory and loads new secure code into it.
2. Momentarily freezes all processes in safe mode.
3. Modifies original functions and jumps to the new secure code, ensuring old vulnerable code does not execute.
4. Unfreezes all processes and resumes system execution.

Conclusion

The longer organizations leave their servers unpatched and out of compliance, the higher the risk of compromise and a data breach. Automated patch management solves the problem of outdated and unpatched software, but it still requires downtime from a reboot. Live patching gives organizations the best of both worlds — rebootless patching with patch management automation. Not only does this save time and administrative overhead, but it can save an organization millions in data breach legal fees, customer reparations, brand damage, and compliance violations.