Securing open source software dependencies in the public cloud

I recently recorded a Lightboard presentation on securing open source software dependencies in the public cloud. This blog summarises, and expands upon, some of the key elements from that presentation:

I think about this topic through two lenses: software supply chains and updating software dependencies while maintaining stability.

Software supply chains

Software supply chains are the process of converting your source code to a software artefact like a package. In most cases, that will incorporate third-party dependencies. The package you create will also likely be a dependency in a later stage of the build process. Every step of this build process introduces a potential vulnerability or an avenue for exploitation.

One example of this was the SolarWinds supply chain attack. SolarWinds had reportedly been installed in numerous critical organisations, including Microsoft and US government departments. SolarWinds stated:

The vulnerability was not evident in the Orion Platform products’ source code but appears to have been inserted during the Orion software build process.

SolarWinds Update on Security Vulnerability, as archived 27 September 2021

The takeaway from this incident is that reviewing the source code is not enough. Even if you have reviewed the source code for security vulnerabilities, you need to ensure that you trust the build process. That means either bringing the source code back into your build environment or having confidence in the build process of the organisation or company that builds it.

Updating software dependencies with stability

When building software, you will nearly always incorporate dependencies to save rewriting everything yourself. In modern code bases, these are very likely open source dependencies. Using open source dependencies is normally a great idea, as many people can share the development effort of the functionality you need. The challenge is that these dependencies are their own separate projects with their own upstream maintainers and release timelines. It is unlikely that the upstream maintainers will keep their project on a release timeline that aligns with your software project.

If one of your dependencies has a security vulnerability, you have three main options:

1. ignore it (which is not normally a real option);
2. allocate sufficient developer time to incorporate the fixes safely; or
3. use Extended Security Maintenance.

Ignore it

In 2017, Equifax relied on the Apache Struts framework. A security vulnerability was found in the framework and Apache Struts released a fixed version. Unfortunately, Equifax did not roll that fixed version out to its production systems in the months that followed. The result was a data breach that exposed nearly 150 million individuals’ personal data and led to a $575m settlement. Ignoring security vulnerabilities in your dependencies is probably the wrong option.

Allocate developer time

A second option is to allocate your development teams sufficient time to solve this problem. They need to monitor the upstream projects for relevant security vulnerabilities and carefully review the fixes to determine the best approach to incorporate them into your software project. The two main approaches there are:

1. pulling in the latest version of the dependency; or
2. vendoring the dependency and trying to backport important security fixes yourself.

Developers will often take the latest version of the dependency to address a security vulnerability. They may pull this from the upstream repository or an ecosystem package (e.g. pip or npm). The challenge, however, is that many open source software projects will mix security fixes, bug fixes and new features in new versions. That may mean that, in addition to the security fix that your team needs, bringing in the latest version also brings in a change to the API or the dependency’s behaviour. This means that you should put the revised combination through quality assurance, and potentially user acceptance tests with your key customers, each time you update a dependency in this way.

An alternative “do it yourself” approach is to bring selected fixes back from the upstream into your codebase. This means your team then carries the maintenance burden of this “vendored” dependency and fixes. It can become a challenge to determine whether or not future vulnerabilities affect your unique, vendored version of that dependency. Generally speaking, the more people using, finding and fixing bugs in open source the better. That means uniqueness also carries stability risks, as nobody else is using the exact same code as you.

It is possible to keep your dependencies up-to-date with sufficient investment of developer time. Investing this time, however, can reduce the time that the team can invest in more valuable activities like new features. There is also the risk that the friction of doing it yourself creates a disincentive to roll out new security fixes to production systems.

Expanded Security Maintenance (ESM)

Canonical provides a useful third option to customers trying to solve this problem, called “Expanded Security Maintenance” or “ESM”. ESM gives developers approximately 30,000 packages in the main and universe repositories of Ubuntu. These packages span the full spectrum of tools, language ecosystem packages and libraries that developers build on.

Each time a high or critical security vulnerability emerges in the 10 years following release, Canonical’s security team will create a patched package. They will put that fix back through the same trusted build process that you are using for all of your other Ubuntu packages, meaning that you are not increasing your exposure to software supply chain vulnerabilities. They will also run some tests on the package and ultimately produce a drop-in replacement package for the dependency you are using.

This replacement package will contain just the security fix, maintaining API stability and not changing functionality. This makes it easier for your team to take the updated combination through quality assurance and user acceptance testing. Importantly, fewer changes and easier testing means you can push the fix into production as quickly as possible.

This approach becomes even more valuable the longer that your software stays in production. That is because the difference between your version of the dependency and the latest (security-patched) upstream will grow. There are likely to be a lot of changes to a dependency in 10 years. ESM gives you a unified security maintenance commitment, for the same set of packages and versions, for the same period of time.

Obtaining ESM in the public clouds

ESM is available as part of Ubuntu Pro. For datacenter use cases, Canonical makes ESM available with Ubuntu Pro tokens that you attach in each machine or instance. We have worked with the three biggest public clouds, however, to make using Ubuntu Pro as easy as possible. Ubuntu Pro on these clouds will launch with entitlements attached automatically and ESM turned on by default out-of-the-box. These are also billed via the public cloud, avoiding the need for a separate procurement relationship. See how to launch these or upgrade your instances to Ubuntu Pro on AWS, Azure or Google Cloud Platform. Oracle or IBM Cloud users should contact us.

Conclusion

Creating stable and secure software is about more than just the code that your developers write. You need to be confident that the build system that your code uses meets your requirements for security. If you are using dependencies, open source or otherwise, your team needs to have a plan for when there is inevitably a vulnerability in those dependencies. Your team needs to ensure that it either budgets the time to incorporate security fixes into your software, with the security and stability requirements that you have, or perhaps you should take a look at ESM.

Further reading:

Learn how to take advantage of ESM

A guide to open source vulnerability management

Ubuntu updates, releases and repositories explained

Ubuntu Pro FAQ