Introducing Bare Metal Kubernetes: what you need to know

Bare metal Kubernetes means deploying Kubernetes clusters and their containers directly on physical servers, instead of inside traditional virtual machines (VMs) managed by a hypervisor layer.

Bare metal Kubernetes can be used in edge computing deployments to avoid the overhead of running a VM layer on small form-factor hardware, or in the data center to reduce costs, improve application workload performance, and avoid the cost of VM licensing.

As we’ll discuss, deploying on bare metal has historically been challenging. Spectro Cloud research found that only 20% of enterprise Kubernetes adopters use bare metal.

But with the help of tools like Cluster API and Canonical MaaS, provisioning bare-metal K8s clusters is now much easier, more scalable and efficient.

As a result, we are seeing more enterprise operations teams starting to use bare metal in their Kubernetes deployments.

Keep reading for everything you need to know about bare-metal Kubernetes, including how it works, how it compares to VMs, and how to get started with your own bare metal deployment.

Background: what is Kubernetes?

Before diving into the details of bare-metal Kubernetes, let's start with a short primer on what Kubernetes does.

Kubernetes is an open source container orchestration engine, and it’s by far the most widely used, outpacing alternatives. Research from the Cloud Native Computing Foundation (CNCF) found that 96% of its respondents were either considering or using Kubernetes.

Kubernetes is a container orchestration engine

Kubernetes’ main job is to schedule and deploy containerized applications, one of the key elements of the cloud-native model. Compared to a VM, containers are lightweight because they don’t contain an operating system, and they’re portable because they contain all the application’s dependencies.

You don't strictly need Kubernetes to run containers, as you could manually deploy containers on your servers’ operating system using a container runtime. But that approach isn't practical unless you have just a handful of containers to work with. You need an orchestration engine if you want to scale.

Scaling with clusters and worker nodes

That’s what Kubernetes does. It deploys containers onto ‘pods’ in a cluster of ‘worker nodes’ managed by a single ‘master node’ control plane through the Kubernetes API, and scales as needed.

Kubernetes is extremely powerful, thanks to its scalable architecture. To achieve high availability and load balancing, Kubernetes fires up and tears down as many worker nodes as it needs. It places applications across them depending on policies and user demand.

For example, should a node go down or suffer another issue, Kubernetes can port the workload to other nodes, preventing downtime.

What makes bare metal Kubernetes different?

Kubernetes and containers emerged into a world where VMs were the de facto environment for deploying applications. Naturally, most businesses have been deploying containers and Kubernetes on top of VMs, which in turn sit on top of a hypervisor layer and a host OS on the hardware (let’s not get into the idea of a bare-metal hypervisor…).

Bare metal Kubernetes, by contrast, removes the VM layer and its hypervisor overhead and puts the Kubernetes installation directly on the host server’s operating system (OS).

This means that containers running in a bare-metal Kubernetes cluster have direct access to the underlying server hardware. This would not be the case with clusters composed of virtual machines, which would abstract the containers from the underlying server hardware.

It's worth noting that bare-metal Kubernetes can happen in all kinds of different environments:

• On on-premises servers deployed in a conventional server room or data center
• On a hosted bare metal service in a colocation provider, public cloud or managed service like Hivelocity.
• On a variety of edge hardware, including Raspberry Pis to 1U rackmount servers or small form-factor devices, or telco edge servers in points of presence (PoPs).

“Thus, bare-metal Kubernetes can accommodate any type of cloud architecture or hosting model – on-premises, public cloud, multicloud or hybrid cloud. As long as Kubernetes is deployed directly on the server OS, it’s bare-metal.”

The Advantages of Bare Metal Vs. Virtual Machines

Let’s get one thing clear: from the perspective of Kubernetes itself, the type of servers that exist in a cluster doesn't really matter.

Kubernetes will happily orchestrate your containers regardless of the underlying servers. In fact, Kubernetes is not designed even to recognize the difference between a bare-metal server or a VM, let alone treat them differently.

This doesn't mean, however, that choosing bare-metal servers has no impact on your Kubernetes environment. On the contrary, a bare-metal approach to Kubernetes can deliver several key benefits.

Better performance without a hypervisor layer

For many workloads, bare-metal Kubernetes will result in higher overall performance, for two reasons.

First: because bare-metal clusters don't have a virtualization hypervisor layer (which consumes resources), there are more resources available for workloads to consume. This will typically result in better overall performance, especially for workloads that come close to maxing out total infrastructure capacity.

Second: it allows containers to access bare-metal hardware resources, like GPUs, on host servers. Most applications don't need those resources, but certain types – such as those that use GPU offloading to accelerate data analytics operations – will run much faster in this type of environment.

Reduced licensing costs and complexity

Bare-metal Kubernetes doesn't automatically translate to lower overall costs. Depending on the workloads and hardware setup you’re running, you may see higher asset utilization with either bare metal or virtualized setups, so your hardware costs may go up or down.

But what is clear is that by doing away with the commercial hypervisor layer you can avoid licensing costs and potentially other expenses like support contracts.

In our research, 68% of Kubernetes users said that Kubernetes will ultimately eliminate the need to pay for a hypervisor layer.

You may also see operational efficiencies. In a bare metal world your team still has to provision and manage server hardware and handle things like replacing failed drives, network cards or PSUs (at least in an on-premises or edge scenario). But your team doesn't have to set up or manage hypervisors and VM profiles. In a sense, you’re getting rid of a second orchestration layer and the complexity and abstraction that goes with it.

“Eliminating the VM layer is simply one less technology to learn, manage, govern and maintain.”

Improved cluster security and control

Security has become a top concern among Kubernetes users. Although running Kubernetes on bare metal doesn’t solve all security challenges, it is advantageous in the sense that it gives admins full control over the servers that host your nodes.

You can decide which operating system to run on the servers, which security hardening frameworks (like SELinux or AppArmor) to install and how to configure user accounts on the nodes in order to mitigate security risks.

If you run Kubernetes on VMs that are managed by someone else (like a cloud provider), you wouldn't get any of these controls over the host servers; you'd only be able to control the VMs.

Bare-metal Kubernetes is also more secure in the respect that when each node runs as a standalone server, you don't have to worry about security breaches on physical servers affecting multiple nodes. That's not true when you run multiple nodes as VMs on top of a single physical server. In this case a security issue on the underlying server could impact all nodes running on it.

The challenges of bare-metal Kubernetes

While bare-metal Kubernetes offers clear advantages, it also presents some challenges:

• Provisioning: Provisioning your servers may be more difficult. Instead of deploying VM images to set up your servers, you'll need to configure each bare-metal machine using a different approach. Tools like Canonical MAAS or the Tinkerbell Cluster API Provider can help, but the process is still likely to be more complex than setting up VM-based clusters.
• Day 2 operations: We already said that Kubernetes itself doesn’t care about the server hardware and OS that sits underneath it. Without a virtualization layer to manage the hardware and the operating system, how do you manage the OS and perform day 2 operations tasks like OS updates and patching?
• Backup and migration: For similar reasons, backing up bare-metal servers, or migrating them to different hardware, can be more challenging. You can't simply take VM snapshots or perform image-based backups.
• Node failure: When you run each node as a standalone server, you are essentially putting many of your "eggs" ( or containers) in one "basket" (or server). For example, the failure of one node (due to an issue like an operating system kernel panic) would bring down all containers hosted on that node. With virtual machines (VMs), you could "slice and dice" a single bare-metal server into multiple nodes. So in the event of a node failure, the remaining nodes (and containers hosted on the underlying server) will continue to function.

Is bare-metal K8s right for you?

Bare-metal Kubernetes is a strategy that lets teams take full advantage of bare-metal infrastructure for hosting containerized workloads. We stand by our prediction that it’s going to grow in popularity. And we’re already seeing some big companies like T-Mobile excited by the results they’re getting. But is it right for your Kubernetes use case?

To answer that question, you must determine whether the advantages outweigh the drawbacks, based on workload requirements and your team's level of expertise.

If your engineers are accustomed to working with bare-metal hardware and can simplify the provisioning and management processes – a bare-metal approach may be a good fit for you. That's all the more true if your workloads can benefit in particular ways from access to bare-metal infrastructure, like direct access to GPUs.

And, if you need help planning, implementing or managing bare-metal Kubernetes clusters, take a look at five best practices in this blog.