.svg){kind=logo}
We’ve been writing about multicloud for years. Now, in 2024, multi-cloud has graduated from the “early adopters” phase into the “early majority” phase in the technology adoption lifecycle.
While survey results are often influenced by selection bias, various resources ranging from the Enterprise Cloud Index by Nutanix, HashiCorp State of Cloud Strategy, and Infrastructure as a Service Market Report from HG Insights all point to 30-80% multi- or hybrid-cloud adoption, with higher numbers in enterprises than smaller startups.
Kubernetes is no exception in this trend. Spectro Cloud research has found that most enterprises already run multiple Kubernetes clusters, and use multiple clouds.
As companies adopt a multi-cloud strategy, infrastructure teams must build, deploy, and manage Kubernetes clusters across AWS, Azure and Google Cloud (GCP), and even on-prem options such as VMWare or OpenStack.
Although Kubernetes provides a more standardized interface across clouds, managing multi-cloud Kubernetes is no simple feat still in 2024.
Let’s take a deeper look at what exactly we mean by multi-cloud Kubernetes, some of the reasons for pursuing it, as well as some challenges and different options for implementing it.
What is multi-cloud Kubernetes?
In a nutshell, we’re talking about running Kubernetes across multiple clouds.
There are largely two architectures for companies implementing multi-cloud Kubernetes:
- Having a single cluster that spans across multiple clouds (e.g., running master nodes for the Kubernetes control plane on AWS, which manages worker nodes on AWS and Azure).
- Having multiple clusters across multiple clouds, but connecting them through other means (e.g., service mesh) to abstract away the infrastructure difference while taking advantage of different cloud offerings (e.g., AI on GCP vs. confidential computing on Azure).
While option 1 technically works and still counts as multi-cloud Kubernetes architecture, the world has largely settled on option 2 as the preferred multi-cloud Kubernetes pattern. This is because option 1 suffers from latency as well as hefty ingress/egress charges across clouds.
The multi-cluster Kubernetes setup in option 2 will inevitably also incur some of those problems, but by colocating workloads by use case, you can deal with those challenges more effectively.
It’s important to also note that multi-cloud Kubernetes encompasses both managed offerings like EKS, AKS, and GKE, as well as self-hosted options such as k3s or MicroK8s. While most enterprises leverage popular offerings from hyperscalers, multi-cloud Kubernetes extend to usage on smaller clouds like Hetzler, Linode, and Digital Ocean as well. If your company has edge or on-prem needs, multi-cloud architecture may cross-over with hybrid-cloud architectures too.
What’s driving adoption?
Adoption of multicloud Kubernetes deployments is a natural by-product of the wider multi-cloud trend. But there are several possible benefits of multi cloud Kubernetes too, including:
- Vendor risk: distributing workloads across multiple clouds protects against vendor outages and disaster recovery scenarios like the one from earlier this year where Google Cloud accidentally deleted Australian fund UniSuper’s account. Data was only restored from a tertiary backup stored on another cloud.
- Security and compliance: some companies are subject to specific regulatory rules that force operating on multiple clouds (e.g., finance sector) or they might be subject to various laws in different regions of the world (e.g., China).
- Cost management: cloud providers compete by offering competitive pricing on certain services or usage. Depending on the use case, it may be more economical to distribute workloads across clouds based on contract terms.
- Flexibility: ultimately multi-cloud allows companies to pick and choose the best of each of the cloud providers. For example, they may use a certain cloud for their AI offerings, while running some of their other workloads based on the cheapest price or availability (e.g., GPUs, ARM instances).
Since Kubernetes acts as a de facto standardization layer for workload portability, it has actually accelerated the multi-cloud trend. Companies are now able to implement multi-cloud strategies easier than before through Kubernetes.
Challenges with multi-cloud Kubernetes
Just because Kubernetes makes multi-cloud easier doesn’t mean that it is easy — as we have discussed at length in previous blogs. Anyone who has dealt with cloud infrastructure knows that while there may be feature parity across clouds, it still takes work to manage each multi-cloud environment in a consistent manner. And this holds true both for conventional IaaS cloud services, and the supposedly managed control planes you get with services like EKS or GKE.
For example, different cloud providers deal with project/account organization and identity access management slightly differently. Also, certain functionalities may be bundled in one single product vs. split up into multiple options across clouds (e.g., GCP Pub/Sub vs. AWS SQS + SNS).
Even within Kubernetes, there are subtle differences between cloud offerings. Some clouds manage the lifecycle of the nodes for you, whereas others make that the responsibility of the cluster admin.
Since identity access management (IAM) differs across clouds, inevitably mapping Kubernetes roles with cloud roles to interact with other cloud services (e.g., accessing RDS from EKS vs. CloudSQL from GKE) differ in syntax and implementation. Abstracting away the complexities across clouds and keeping up with the new features is no easy feat.
And we haven’t even talked about Day 2 concerns like observability, security, and node upgrades and maintenance.
Yes, managed offerings from different clouds solved these to a certain degree, but when running a multi-cloud setup, we want a single, unified experience to control these in a single manner.
Ideally, this should extend beyond just Kubernetes management as there are likely workloads outside of Kubernetes to keep in mind.
Approaches to managing multi-cloud Kubernetes
Whatever cloud platforms you use for your multi cloud Kubernetes clusters, your ideal end state should be centralized management. Otherwise, you’re building silos and increasing your management overhead.
There are severals ways to manage multi-cloud Kubernetes effectively, each with their pros and cons:
Building on top of IaC tools
Many companies already manage their cloud infrastructure via infrastructure as code (IaC) tools like Terraform. There are extensions built on top of these tools, like Terragrunt, to make multi-cloud setup a bit easier to manage and scale.
Others might be using a more programmatic approach via CDKTF or Pulumi to leverage programming languages to define infrastructure. Simply extending existing IaC tooling would be the fastest way to achieve multi-cloud Kubernetes.
But in the end, there will either need to be some abstraction layer built on top to manage the underlying Terraform modules or somehow manage each of these different cloud offerings separately.
Also, unless Kubernetes is fully-managed by these IaC tools, yet another tool must manage Kubernetes setup including identity, workload, and node lifecycle.
Using Kubernetes-based tools
The other approach is to leverage a cloud-native control plane framework like Crossplane to define cloud infrastructure as if it is a Kubernetes workload. The promise of Crossplane is to manage everything via Kubernetes, so that you are left to simply handle Kubernetes setup across clouds.
However, Crossplane does not handle all cloud workloads as well as current IaC tools and has limited support for smaller cloud offerings. Also, for teams with existing IaC tooling or lots of non-Kubernetes workloads, migrating to this pattern might be a heavy lift.
Enterprise Kubernetes management platforms
There are multi-cloud Kubernetes management offerings from hyperscalers (like Google Anthos and Azure Arc) that allow you to run their version of managed Kubernetes in other clouds. It is unclear how much traction this has gotten, but the one clear disadvantage of this approach is that vendor risk still exists to a certain degree.
Outside of those offerings, there are many options for enterprise Kubernetes management platforms including legacy products such as Rancher, OpenShift and VMware Tanzu.
Many of these providers offer a scalable platform that can manage large numbers of clusters with a centralized dashboard with observability, security, and management built-in. But it is also important to consider the complexity, cost, and yet another form of vendor lock-in when considering these options.
Enter Palette by Spectro Cloud
Palette is a complete integrated multi-cloud Kubernetes management platform by Spectro Cloud that combines the best ideas from the approaches described above.
As an enterprise-ready Kubernetes management platform, Palette helps customers adopt and manage multi-cloud Kubernetes at scale. Palette supports not only public clouds including AWS, Azure, and GCP, but also Tencent TKE.
Support for these clouds include both IaaS and managed Kubernetes services, and GovCloud versions for both AWS and Azure, and spot instances to help bring cloud costs down.
Hybrid-cloud scenarios are also supported for VMware, Nutanix, OpenStack, and even bare metal use cases via Canonical MAAS.
Secondly, Palette provides full-stack management for various Kubernetes clusters. It has an extensive list of Kubernetes integrations and allows users to specify different OS, container network interfaces (CNI), container storage interfaces (CSI), and other add-ons, unlike other platform providers with rigid opinions. Palette packages the entire stack ranging from the underlying infrastructure for Kubernetes as well as Kubernetes components and manages them as a single unit.
Finally, Palette also offers comprehensive end-to-end declarative lifecycle management ranging from day 0 design, to day 1 provisioning, and day 2 operations (e.g., security patches, upgrading nodes, upgrading Kubernetes versions).
Underneath the hood, this is possible because it is leveraging the Cloud Native Computing Foundation (CNCF) open-source Cluster API (CAPI) project. CAPI provides declarative APIs to create and manage Kubernetes clusters the way Kubernetes creates and defines its own manifests like pods and configmaps.
Palette has been extended further by CDW as part of its Multicloud GitOps Foundation (MGF) offer, which brings together Palette’s declarative Kubernetes orchestration with CI/CD pipelines and Terraform Cloud for IaC. It completes the orchestration picture.
Wrapping up
Multi-cloud adoption will continue to accelerate in 2024, becoming almost a starting point for all cloud architecture. As the recent Google Cloud outage has shown, there is real, inherent risk in relying on a single vendor, no matter how small the possibility is.
With Kubernetes providing a standardization layer across clouds, more teams are now looking to multi-cloud Kubernetes architectures to protect against such risks.
Yet in 2024, implementing and maintaining multi-cloud Kubernetes is still not as simple as it sounds. There are differences across clouds that traditional IaC, Kubernetes tooling, or other management platforms do not solve very well.
Palette is a modern platform that leverages the core concepts from Kubernetes to define and manage other clusters like any Kubernetes resource or object. This helps bridge the gaps in current challenges with multi-cloud Kubernetes management in an elegant manner, enabling multi cloud deployments to be managed from a single interface, with consistent operational patterns.
If you’d like to learn more about Palette, check out our docs, or set up a personal demo to get free access to try it for yourself.
