Selecting the best Linux distribution for Kubernetes clusters

With the hope of avoiding the holy war that revolves around the various Linux distros, this blog article seeks to run through some of the advantages (and disadvantages) that Linux distros bring to the table when you have the goal of installing a K8s cluster. Some highly established options were considered along with some new kids on the block. It is assumed that you have a completely free choice in the Linux distro that you use. If this assumption is wrong then you probably don’t need to read further anyway. Moving briskly on, here are our contenders:

• Red Hat (RHEL/CentOS)
• Ubuntu 14.04
• TinyCore Linux
• Clear Linux
• RancherOS

Starting with the old boys,

RHEL/Cent OS and Ubuntu have similar things going for them. Given that they are both well established, and frequently used, distros you can rely on the base OS being well tested and that there will be a great number of Docker installations based on them. This will mean that when you run into problems setting up and/or running your K8s cluster, Google will deliver many solutions relevant to your environment. They also have well-polished installation tools. Simply waving the installation ISO near a server will usually result in a successful install. Post installation, tools and configuration will generally be where you expect them and the package managers will be familiar to most Linux system administrators out there.

Docker is directly available in those package managers, and newer versions can be easily added from additional repositories. RHEL is even officially supported software, although you have to consider how helpful that support will be given the end goal is largely to install Docker tools on top of a kernel. This leads us nicely on to the disadvantages of these older distributions. Both are generalists and deliver far more than is needed to set up a K8s cluster. Although not a problem as such, both require a little extra work to bolt down those extras so that we don’t end up maintaining components that we aren’t planning to use, but nevertheless act as potential attack vectors into our new K8s cluster. In particular, a feature that RHEL/CentOS delivers, namely SELinux, gets in the way of a K8s installation and is usually the first thing (other than swap) that needs to be disabled.

If it bloat that we wish to avoid,

the next two contenders come gallantly to our aid… TinyCore Linux and Clear Linux are both designed to deliver much leaner installations. The former of these will be familiar to a good cross section of readers, since it is very commonly used as the operating system of hardware appliances (e.g. firewalls, NAS, etc.). The main advantage of TinyCore, and the reason that it is commonly used in embedded environments, is that it has a really small footprint. The other good news is that Docker does run on it. Sadly, the bad news that you need to compile Docker (and recompile the kernel) yourself follows right on afterwards. In other words, Docker can be persuaded to run on TinyCore. Clear Linux, being a niche distro, contains packages that mostly target various scientific groups. One of these groups happens to be container management so it looks very promising to us.

After running through the base installation, and installing Docker with:

sudo swupd update

sudo swupd bundle-add containers-basic

sudo systemctl start docker

the resulting system contains systemd (and some associated daemons e.g. for time synchronisation), docker, pacrunner (network manager proxy) and nothing else. Further to this minimalistic installation, there is also support for Kata Containers which enhance container security to the level of virtual machines. This all looks great, but there has to be a catch, right? Well, half a catch depending on who you ask. Long story short, auto update is enabled by default. Yes, it can be easily disabled and it certainly has its merits, but for end users with a rigid, or better said “well defined”, versioning policy it definitely needs to be kept in mind. One other little niggle is that the keyboard layout selected during the installation was not actually applied to the system. Although this is inconvenient to us non Americans, it can be worked around, nevertheless, it does hint towards a less well rounded setup program. Niggles aside and assuming we don’t have a problem with our operating system automagically updating itself, we have surely found our winner, right?

Well…

Remember that I previously mentioned that our goal was to have Docker tools running on a Linux kernel? As luck would have it, that exact thing exists and it is called RancherOS. This distribution replaces the init process (pid 1) with a Docker daemon that runs all system services in containers and a further Docker container where user land containers reside. Thus:

How to install this magic animal you ask? In my case, since I happened to have docker-machine and virtual box installed on my laptop, I simply needed to issue the command:

docker-machine create -d virtualbox

--virtualbox-boot2docker-url https://releases.rancher.com/os/latest/rancheros.iso

--virtualbox-memory

The resulting installed system is as minimalistic as can be achieved and directly ready to run Docker containers, or have K8s installed on it. When working in an infrastructure as code environment this distribution would certainly make our lives a little easier. Another argument for RancherOS is the existence of Rancher Kubernetes Engine (RKE). This tool allows you to define where particular components of a K8s cluster should be located and then easily provision (and later update) the defined infrastructure. True, this useful tool supports most Linux distributions, and the documentation also states that Ubuntu was used heavily during its development, but one has to imagine that future testing would concentrate on the RancherOS sister product.

In conclusion,

I am personally torn between my familiarity with Ubuntu and the promise of RancherOS. My previously successes with K8s setup have been when provisioning up a collection of Ubuntu servers and then throwing kubespray (an Ansible based K8s installer) at them. However, I find myself intrigued by RKE and RancherOS and plan to test these out in the future. Sounds like I have another blog article to write sometime.