Taking a stand against unofficial Ubuntu images

Ubuntu is amazing on the cloud because we work with cloud providers to ensure crisp, consistent and secure images which you can auto-update safely. On every major cloud—AWS, Azure, Google, Rackspace, SoftLayer and many more—you can be confident that ‘Ubuntu’ is Ubuntu, with the same commitment to quality that you can expect when you install it yourself, and we can guarantee that to you because we require that clouds offer only certified Ubuntu images.

Often we optimize performance or integrate with underlying cloud requirements, and a custom image for that cloud is produced in partnership with their engineers. We make the same certification commitment to those customized images that we do to the public release of Ubuntu. This provides a universal promise of performance, stability and security while imposing no premium cost on users of those images. You as the user get it free, but it takes a lot of work to get it right, and Canonical holds those pieces together, in partnership with the clouds.

We are currently in dispute with a European cloud provider which has breached its contract and is publishing insecure, broken images of Ubuntu despite many months of coaxing to do it properly. The home-grown images on the cloud, VPS and bare metal services of this provider disable fundamental security mechanisms and modify the system in ways that are unsupportable. They are likely to behave unpredictably on update in weirdly creative and mysterious ways (the internet is full of fun examples). We hear about these issues all the time, because users assume there is a problem with Ubuntu on that cloud; users expect that ‘all things that claim to be Ubuntu are genuine’, and they have a right to expect that.

We have spent many months of back and forth in which we unsuccessfully tried to establish the same operational framework on this cloud that already exists on tens of clouds around the world. We have on multiple occasions been promised it will be rectified to no avail.

We are now ready to take legal steps to remove these images. We will seek to avoid affecting existing running users, but we must act to prevent future users from being misled.

We do not make this move lightly, but have come to the view that the value of Ubuntu to its users rests on these commitments to security, quality and updates.

It is my hope that these steps bring the matter to the attention of management at this cloud who can make a binding commitment to doing it properly, fixing the problem, but they may also choose not to have Ubuntu. That’s OK. Thousands of developers participate in making Ubuntu, they do not expect to see that work tarnished for lack of commitment on the part of corporations who make no contribution to the effort.

More importantly, as users, you can vote with your feet to any significant cloud and have confidence that what you are using won’t bite you hard in the weekend.

In the same way that pre-installs of Ubuntu need to be certified equivalent to what you download, cloud providers who offer an Ubuntu experience for devops must participate in this program. I’ve heard enough hollow promises from vendors and clouds (‘trust us we can do this just fine’) to know that, without this program, it would be a total mess out there.

To count some of the ways we have seen home-grown images create operational and security nightmares for users: clouds have baked private keys into their public images, so that any user could SSH into any machine; clouds have made changes that then blocked security updates for over a week; clouds have confused users with image- or kernel-soup, and users have been pushed into building their own images; VMs have had changes that resulted in very slow boot or poor performance; unstable kernels that disable features Ubuntu packages expect to be there; and many more. When things like this happen, users are left feeling let down. As the company behind Ubuntu, it falls to Canonical to take action.

Ubuntu is the leading cloud OS, running most workloads in public clouds today, thanks to its security, versatility, and policy of regular updates. It underpins the services of some of the most exciting cloud-native companies such as Netflix, Airbnb, Heroku and many more. I love the underdog, and Ubuntu is also something of an underdog. It is the only free cloud operating system with the option of enterprise-grade commercial support and government certification. We take the responsibility very seriously, and we invest in tools and capabilities for Ubuntu users on those clouds that improve their performance and resilience.

Our public cloud programme delivers critical security updates against all supported packages and addresses kernel vulnerabilities such as Dirty COW and its precursors. Certified clouds received regular fresh images and same-day in-region patches, for every security update since 2013. That minimises bandwidth costs and downtime for Ubuntu users running on those clouds.

On most clouds you can also use the Livepatch service to avoid reboots—useful for container hosts, for example. Livepatch users had a patch for Dirty COW streamed to them and applied without rebooting within hours of the vulnerability becoming published.

I doubt this company would have done what they did with RHEL or Windows, and the Ubuntu community deserves just as much respect for its product, rights and reputation. We have the same rights with regard to Ubuntu, we just choose to make Ubuntu widely available at no cost. That does not entitle any provider to jeopardise the image or the brand. Our policy has been acknowledged by dozens of clouds, and third party community and open-source centric organizations like the Free Software Foundation and Software Freedom Law Centre. We care about quality, security, maintenance, updates and upgrades, and users trust us because we care, so we will defend that commitment.