An overview of live kernel patching

Live kernel patching is an essential component of your Linux server management and vulnerability remediation strategy. Patching encompasses all installed software, and software essential for baseline operation of the OS. The most dynamic component of Linux, essential for baseline operation, is the kernel.

The Linux kernel has over 30 million lines of code, and in January 2022, the stable Linux kernel version had over 999 commits. It is not uncommon for a stable Linux kernel to receive 25 patches every day. Many of these patches address security vulnerabilities, which may or may not be formally disclosed. System Administrators need to insulate their estate from critical vulnerability exploits, which target one of the most scrutinised pieces of software in history.

Ubuntu offers many mechanisms to improve security. For example, AppArmor is a Linux security module (LSM) native to the Linux kernel. AppArmor reduces the kernel’s attack surface and improves security through application confinement. Ubuntu 20.04 introduced support for an optional kernel lockdown mode LSM, and also shipped with Kernel Self-Protection. However, one of the most significant mechanisms to safeguard Ubuntu’s kernel was introduced in 2016: live kernel patching.

A completely unpatched kernel provides the illusion of stability. Unpatched kernels put your data at risk through potential privilege escalation attacks, or put your service uptime at risk through denial of service vulnerabilities. The cost of a successful attack far outweighs the costs of managing Linux servers responsibly.

What is live kernel patching?

Live kernel patching is an interim solution for high and critical severity kernel vulnerabilities, which need to be addressed immediately without any interruption of service, on high availability or mission critical workloads.

The kernel live patch process patches the Linux kernel while the system runs. Kernel live patching inoculates your server in real time, without requiring a reboot. By default, the root user has the CAP_SYS_MODULE capability. This privilege is required to modprobe a module into the Linux kernel. Canonical’s Livepatch solution leverages ftrace, and aligns with the upstream Linux kernel’s live patching technology. Canonical Livepatch compiles a new function which addresses a security vulnerability, and relies on ftrace to redirect calls to this new, patched function.

Kernel live patching is, unsurprisingly, limited to just the kernel. It is not possible to live patch software packages installed through package managers this way.

To reboot, or not to reboot? When, is the question.

Anyone can file a formal security vulnerability report, in the form of a Common Vulnerability and Exposures (CVE) record at cve.org. This central repository is an internationally recognised catalogue of vulnerabilities, with almost 169,000 reports. A small fraction of kernel security fixes are documented through the CVE process, and some CVEs have undocumented follow-on fixes. Therefore, a kernel restricted to CVE patches only, may still pose a security risk.

At the next scheduled maintenance window, machines with live patched kernels should be fully patched. Fully patching a system would require a sudo apt update; sudo apt upgrade -y followed by a reboot. This activity will patch any missing kernel updates that were not associated with a CVE, and also patch software installed through apt. Landscape offers a choice between a fully-automated upgrade solution and a point-and-click experience to manually trigger the upgrade, over any cross section of your Ubuntu estate.

Chart kernel live patching over time with Landscape

Canonical’s live kernel patching solution, known as Livepatch, works seamlessly alongside Landscape. Landscape is Canonical’s Linux server management product.

Linux servers with live patched kernels need to be rebooted at the next earliest maintenance window. Charts in Landscape provide a visual indication for how long a server has been live patched, and is awaiting a reboot.

In this example, you can see that every server is completely live patched with the exception of one machine, which has benefited from 48 kernel live patches, and has not yet been rebooted.

If you have 5 minutes and an appetite for custom graphs, follow this short tutorial to add kernel live patch activity on your Landscape dashboard.

Elevate your patch management game

Various teams at Canonical evaluate security vulnerabilities, and prioritise remediations every day. Their work is consumed by a global audience of Ubuntu users ranging from individuals, organisations of all sizes, and governments. Canonical’s globally-distributed teams approach security vulnerabilities as a continuous workstream, oftentimes proactively addressing security concerns before formal reports are even filed. When evaluating criteria to find the best linux server management dashboard for your organisation, consider which products benefit from Canonical’s depth of experience, and consume Ubuntu Security Notices.

Live kernel patching bolsters your security profile, and thus indirectly also benefits the stability of your machine. Livepatch on-premises offers granular control over kernel live patch deployments. SecOps teams in organisations with rigorous change and release management policies rely on these granular controls to follow their best practises guidelines, and meet their SLA obligations.

Compare the differences between Landscape SaaS and self-hosted Landscape. When you are ready, follow the self-hosted Landscape quickstart guide, or sign up for Ubuntu Pro, and bolster security across your Ubuntu estate with kernel live patching.