the big idea

In 2003 a mega nerd published a paper titled Integrating Software Construction and Software Deployment. In this paper, Eelco Dolstra identifies all the things wrong with packages from the perspective of the software author that I mentioned previously. I didn’t come up with all of these ideas myself, alas.

He describes a better build system: Maak. Maak is an atomic build system that can directly correlate, from an installed package, exactly which sources contributed to its build. This included dependecy management, and conveniently sidestepped the issues of package scripts, but did include some functional programming interfaces for describing system changes required to successfully deploy some software.

You probably don’t know anything about Maak or what people were thinking about packaging systems in 2003.

A year later he released Maak’s replacement, which you just maybe have heard of. It’s called Nix. The introduction of the paper introducing Nix captures the essence of atomicity in software packaging and distribution so well that I’ll just go ahead and quote it verbatim:

Software deployment is the act of transferring software to the environment where it is to be used. This is a deceivingly hard problem: a number of requirements make effective software deployment difficult in practice, as most current systems fail to be sufficiently safe and flexible.

The main safety issue that a software deployment system must address is consistency: no deployment action should bring the set of installed software components into an inconsistent state. For instance, an installed component should never be able to refer to any component not present in the system; and upgrading or removing components should not break other components or running programs, e.g., by overwriting the files of those components. In particular,it should be possible to have multiple versions and variants of a component installed at the same time. No duplicate components should be installed: if two components have a shared dependency,that dependency should be stored exactly once.

— Nix: A Safe and Policy-Free System for Software Deployment 1

In short: It is very common to use practices like updates of programs on disk while they are running, and to use dynamic linking (which I covered briefly before also). This seems sensible at runtime, but it’s crazy at update time, and this causes all kinds of interesting problems that real people have faced in the real world.

Have you ever updated your operating system - I don’t care which one - while it was running and had things start breaking? Have you ever had the thing that updates your operating system break while it was running? Have you ever had to reinstall programs or your whole OS to deal with this? I think Eelco had faced this, and it bugged him, so he made Nix. It’s pretty good. I’ve used it off and on and don’t have any major issues with the technology. There’s one thing you’ve got to admit, though:

It’s really tough to use, all the time, for everything. Once you start using it, you get locked into how it does things. Once you start updating small thing, system updates can turn into large compile fests. Binary package distributions for Nix break some of the beauty of what it does in the first place. It’s one of those romantically great ideas that make perfect sense until you try to really live with it.

finding a middle ground

In 2011, some folks working on GNOME got sick of how hard it was to test changes to GNOME core components. One of them, Colin Walters, decided to do something about it. He researched a bunch of ways to make it easier to transactionally swap components out, enable rollbacks, install multiple versions of the same thing, and perform quickly while doing it. In short, he wanted an atomic linux distribution.

Virtual machines, which are a little software/hardware construct that fake an entire computer (or as much of one as is practical), took too long and, especially at the time, made hardware acceleration difficult. This kind of ruled them out for people developing a desktop environment.

git was designed for text files, and he was interested in doing this with binary files. He thought the high level design and organization was good, but some implementation decisions would need revisited for compiled binaries.

And so, he of course looked at Nix.

The NixOS people have a lot of really good ideas, and they’ve definitely thought about the problem space. However, their approach of checksumming all inputs to a package is pretty wacky. I don’t see the point, and moreover it uses gobs of disk space.

— Colin Walters 2

Anyways, instead he built libostree from scratch instead.

and now for something completely different

Also in 2011, a small company named dotCloud launched a “Platform as a Service” offering following a YC incubator. This emerging technology offering allowed you to provide some configuration for where your source code lived, and have the service handle building, serving, and scaling the website it was driving. In order to deliver this service, they built a little piece of technology that allowed them to really rapidly isolate build systems and web server processes. This helped them shorten the time from you pushing to code to your website changes being live, and it helped them scale up tiny units of execution quickly in response to spiky traffic with minimal delay.

The primary technology these outcomes had been built with at the time were Virtual Machines, complete with an isolated operating system that would need configuration and updates to its own packages. Systems for streamlining this, like kickstart and cloud-init, existed already. It still increased the maintenance surface, and added extra complexity and scaling overhead that dotCloud wasn’t prepared to make.

A few constructs already existed inside the Linux kernel to help do this isolation and rapid scaling. Notably, most of them were designed by Google for the work they were doing at the scale they were doing it. The basic primitives are simple enough. cgroups allow isolating trees of related processes as a single unit, and allocating a fixed group of resources for that tree to share. Namespaces allow presenting virtual views of kernel-exposed resources to individual processes, such as scoping the filesystem down to a subdirectory, or presenting virtual network interfaces to a process and preventing it from using the resources outside of its own namespace (like the root filesystem and bare metal network interfaces present on the machine). COW filesystems were being rapidly developed to better reuse shared common “base” filesystems and build on top of them.

Maybe you know where this is heading, but dotCloud combined all these elements behind a single abstraction to build their PaaS and they named the tool Docker. When they built your code, they used a Docker construct that they dubbed the “container.” This term had been in generic use in lots of ways (like in Java), but once Docker was open sourced in 2013, suddenly that’s what everyone meant by the term and all other usage had to move out of the way.

Because Docker was built to run a PaaS where you input source code and the output was a running website, it was built for atomic workflows from the beginning. You input the build context, a process builds an image of everything you need to run the site, you start an instance of that new image, and then you cut over the traffic to the new copy. It didn’t require a new package management format, because the images it were built around were regular OS images - the layering is what gave them atomicity.

This gave you the uptime that the cloud was promising, with build and tooling definitions you already understood. If a build fails, we still have the old version running. If there’s a dependency update somewhere, or an OS package update, you can just build the new image with the same old code you had before. You can test that image as part of a continuous delivery pipeline, to make sure the site will work like you want, before you fire it up and cut that production traffic over. All of the layers, and the manifest of layers that made up a complete image, were content-addressable, and so fully reproducible on other systems (with some fudge factor in there for what a given kernel might do, or how that system is set up outside of the kernel namespace for the container).

Docker containers are smaller than VMs, faster to start, easier for developers to use, and reduced the environmental differences between development and production. In short, they made building and shipping software, as a continuous spectrum, easier and better. A bunch of people liked them very much. Really importantly, since they were made for a PaaS, they were designed around pets vs cattle, making containers resonate as a unit of work for managing fleets of applications on fleets of servers anywhere.

bringing it back

Also in 2013, right on the heels of the open sourcing of Docker, a small group of mega nerds started a project called CoreOS built entirely around containerization. If a Docker Container made it easy to ship server-side software, then CoreOS set out to make it easy to run those containers at scale, to really fully buy into cattle farming. If your applications were just disposable, atomic containers - then your server operating system running those containers should be the same. They designed a distributed key-value store to help keep a consistent set of state for multiple machines, and designed declarative systems like Ignition to handle provisioning and bootstrapping. Real cattle stuff.

A little company called Red Hat was really worried about what the future of server operating systems might look like if those CoreOS guys started taking their business, so they started Project Atomic in early 2014. Project Atomic built, with Colin’s help, on top of libostree with a new layer they called rpm-ostree. Basically, rpm-ostree uses a collection of RPM packages as input to get an atomic OSTree repository out the other side. Don’t be fooled by the CoreOS URL there, by the way - it wasn’t how CoreOS was being built then. But what came next changed more than the GitHub organization rpm-ostree was hosted under, it changed the entire server world maybe more, or at least just as much, as Docker did.

same as it ever was

Google, having implemented most of those kernel primitives that Docker made easy to use, already knew how to run servers as cattle. They’d been doing it in production with a system they called Borg since about 2003, back when Eelco was still just putting pen to paper in college for the ideas that would form Nix.

They quite liked the ideas from CoreOS, especially the simplicity and scalability of etcd. They also obviously liked Docker, since it was just a simple way to interact with the things they were already using. Deciding that clusters of container hosts would be a pretty great way to run software at scale, and getting a bit of an open source itch (well, some of the engineers getting that itch and convincing Google to go along with it), they unleashed Kubernetes upon the world.

By the next month, Docker, CoreOS, and Red Hat (and some others I guess) formally joined the project.

In 2018, Red Hat acquired CoreOS and converted their by-then-Kubernetes-based PaaS, OpenShift, to be native to a blend of CoreOS and Project Atomic technology. OpenShift 4 was the culmination of this rearchitecture.

recap

  • Atomic software package distribution is good for the distributor, developer, and the user
  • The tools that implement it should be ergonomic and easy to use, or else they won’t see massive adoption
  • Containers are ergonomic and easy to use atomic units of software distribution, built for the server
  • These ideas obviously all play well together, all the way from the desktop environment developer up to the distributed computing gurus at major companies operating software at a scale that would make you weep

Guess what bootc does?

  1. Nix: A Safe and Policy-Free System for Software Deployment ↩︎

  2. OSTree README.md ↩︎