Are certs the answer to the K8s skills gap?
Everyone knows that businesses struggle to find platform and devops engineers who really know Kubernetes.
In fact, as we found in our 2024 State of Production Kubernetes research, 75% of organizations say their adoption of Kubernetes has been inhibited by access to expert Kubernetes talent and skills.
And what’s the best way to get the skills, knowledge and competence in a new area like K8s? Training! There are now hundreds of training providers out there, and in response the Cloud Native Computing Foundation (CNCF) offers a series of Kubernetes certifications to formalize what your training should cover.
Certs are super common across the IT industry, from Cisco to Microsoft to the wider Linux Foundation. They’re an easy way to show the world (and by that we mean potential employers) that you are one of those rarest of professionals: a Kubernetes expert, skilled in the latest cloud native technologies.
CKA, KCSA, CKAD… FOMO?!
The most common certification, Certified Kubernetes Administrator (CKA), is undeniably sought after in the job market. It is mentioned on job listings as a requirement for plenty of Kubernetes careers. Over 200,000 people hold a CKA — which sounds like a lot, but by comparison, more than 4 million people have Cisco certifications.
The other CNCF certifications include:
- Kubernetes and Cloud Native Associate (KCNA)
- Kubernetes and Cloud Native Security Associate (KCSA)
- Certified Kubernetes Application Developer (CKAD)
- Certified Kubernetes Security Specialist (CKS)
![types of CNCF certifications](https://cdn.prod.website-files.com/64196dbe03e13c204de1b1c8/66869fa70da7b3a0abda1a19_AD_4nXfjUTFJ1ksq1lZmSjjOK5muKePgLExFKvSoG2usueww4rSpx_iIs4nhagEfLvh4G0m0jUD35qBjVp0OB5vWxx4u5NACc8AyHeAmOYHt4crLasLR5tFsejlHS1reC2MMNnAt1zz-NEWbV-eQ9vc6HiPO5Ivh.png)
The CNCF also has a new-ish “Kubestronaut” badge for those who have earned all five certifications — today only about 150 people hold one.
Many of us here at Spectro Cloud hold CNCF certifications, including all of our support team. My colleague Anton wrote about his experiences with the CKA here. I personally hold the CKA and CKS.
Passing ain’t quick, easy — or cheap
Although there are a ton of free resources out there on YouTube and the internet to learn Kubernetes, many people will naturally choose to take the CNCF’s paid-for virtual and instructor-led training to get ready to pass the exams — or pay for premium content and training from other providers on Udemy or KodeKloud. And don’t forget, it costs money to take these certification exams themselves.
It also takes time. All of the courses take time, as do practice exams. The certification tests themselves can be quite difficult, due to the hands-on nature of the exam and the strict test environment requirements. The exams are online proctored performance based tests and a lot of people complain about the software used and the strictness of the proctoring.
While I found both the CKA and CKS to be difficult for different reasons, I think the CKS exam was overall the harder of the two due to the more specialized knowledge and amount of time each exam objective takes to complete. And I’m not the only one — due to the amount of interest that we’ve seen, Spectro has in the past offered a workshop to help people preparing for the CKS with Palette.
And one more thing: your certifications expire, now after two years (it used to be three). So you need to repeat the process to stay up to date.
Do you really need the CKA?
All this has led many longstanding experts to ask whether it’s worth the effort of renewing their certifications — and many newbies are also wondering whether to go through the process when they’re starting to focus on Kubernetes for the first time. You find the discussions all over Reddit.
When I started working on reviewing the material to renew my own recently expired CKA certification, I started to notice two things.
First: there is so much material in the CKA that I have never had to touch in the entire five years or so that I’ve been working with Kubernetes in the real world.
Second: a load of concepts and activities that I rely on to manage Kubernetes clusters day-to-day in the real world were never covered at all in the CKA.
![What the CKA teaches you](https://cdn.prod.website-files.com/64196dbe03e13c204de1b1c8/6686a029cf2736c5117bce44_AD_4nXcyBv43eaDI1-TTvU0AxKZkwu3s8kKKy7bDSs5sStxWor3GVuDDKseha4Z8Zvv6gQSqci-Nkt4h41QqiQDUxvUkC0oumbtUR5OR9EbIgqjjFly8ysq04yyJjdnw9ZLABV3x_Lq-5drz_8IIElm-ZJWJ2Oy7.png)
What’s missing from the CKA?
Here are just a few examples:
- We are left unprepared to work with an authentication solution like an OIDC provider.
- We aren’t taught how to implement or interact with a container image registry.
- Helm charts are not brought up at all.
These are all things that are necessary for Kubernetes administration in the modern landscape.
I don’t intend to claim or imply that CKA or the other CNCF exams are unimportant or unnecessary for verifying that you understand the fundamentals of Kubernetes.
However, the material does not cover the full scope of what a Kubernetes admin will need to address their day-to-day challenges, working with multiple clusters, using a modern declarative Kubernetes platform.
Let’s take a look at how some of the fundamental topics have evolved, leaving the CKA curriculum behind.
Configuration management
Modern Kubernetes solutions have reached a point where the core components almost never need to be touched.
Previously, if you needed to change configurations for static components such as the kube-scheduler, kube-apiserver, kube-proxy, or etcd, you would need to edit manifest files in /etc/kubernetes/manifests on each individual node, or implement a configuration management system that will maintain the files for you.
With K8s distros that use Cluster API, however, all Kubernetes component configuration is done declaratively in the definition of the cluster object. For example, Palette Extended Kubernetes (PXK) has Kubernetes component configuration values documented as yaml files in its profile definition.
This top-level configuration management is an important evolution for repeatability, ease of troubleshooting, and reliability testing.
For an organization that has multiple clusters (and that is now 100% the norm), or a need to rapidly create multiple more, it makes much more sense to be able to make a configuration change in one place.
Problems related to the configuration are much simpler to resolve, and you will know that every cluster in your environment has the same working configuration — no snowflakes, no configuration drift.
One-off clusters can multiply the complexity of trying to troubleshoot a problem and create a nightmare. You can read more about the importance of this in the Cluster API Core Goals and Design Principles.
As a side effect of this repeatable change management and the maturity of Kubernetes itself, the stability of these components has also grown. The only core Kubernetes component that I have had to troubleshoot in recent years is Etcd, which can have inherent limitations simply due to it being a distributed database (for more information on the inherent limitations of distributed databases, look into the rabbit hole of CAP Theorem).
Authentication
In the CKA curriculum, you spend a lot of time creating Certificates, Keys, Certificate Signing Requests, Roles, and RoleBindings to grant access to specific resources manually.
The concepts are important to understand for a K8s operator, but it is another example of something that you would rarely have to touch when using an enterprise Kubernetes management platform.
Using Palette as an example, user creation and authentication is handled by the identity provider you configure, and you can manage Role Based Access Control via the Tenant admin GUI.
In addition, the scope of access controls can grow far beyond cluster objects in the modern landscape. In an enterprise environment with multiple clusters, there is often a need to define much more robust access controls at a broader level: defining permissions for clusters themselves or specific resources across multiple clusters. In Palette, you can implement these controls at the Tenant- or project-level.
Service accounts are still important if you intend to write your own Helm charts for your own workloads; however, if you intend to use or modify existing off-the-shelf charts, the creation of those service accounts will generally be handled already.
Upgrades
With Kubernetes stacks that leverage Cluster API, performing standard Kubeadm upgrades like you’re taught in the CKA is also an outdated concept.
Rather than having to drain one node, upgrade each individual component, restart systemd units, uncordon the node, and move onto the next (one of the most time-consuming parts of the CKA exam!), Cluster API will simply add a new node with a MachineTemplate already containing the new versions of those components, drain/remove a node with the old version, and repeat the process until every node is replaced with an upgraded one.
This is an improved approach because it allows you to treat a “machine” as an immutable object. This allows for much more stability and security to each individual machine as changes cannot be made directly. If a change is made, it must be made declaratively at the top level, and new machines containing those changes will replace the existing ones.
This is also a more cloud-native approach, as it allows for greater simplicity of scaling. If your “machine” is a clearly defined object at the top-level, then increasing the replicas becomes as simple as increasing the amount of pods in a deployment. As a result, it becomes more fault-tolerant as machines with problems will be automatically destroyed and replaced.
More information and examples about how Palette can improve the upgrade experience can be found in this article.
So what’s the answer?
These few examples prove the point that the standard, widely-used Kubernetes training content doesn’t suffice to prepare you for how enterprises operate Kubernetes today at scale in modern application environments. The tools that are commonly used now have mostly obsoleted many of the concepts that are required for the CKA exam.
I’m not arguing that the CKA is useless. As engineers we all benefit from understanding fundamental concepts of how technologies work, and when things go wrong (and they do!), that understanding can help us work around the unexpected.
But I will say this: if you’re new to Kubernetes, don’t expect that passing the CKA is the end of your journey — it’s the start of a whole new learning curve. And if you’re like me, holding a certification that’s about to expire, maybe think about whether your time and money is best spent on recertifying, or exploring the new frontiers of Kubernetes.
One last point: as we discussed above, modern Kubernetes management platforms automate and abstract away many of the manual, painful tasks you’ll have to perform during the CKA. If you’d like to see one in action, book a demo of Palette and see how you can try it for free.