.svg){kind=logo}
Switching between K8s clusters, using KUBECONFIG, separate config files, or a combination thereof.
If your workflow is anything like mine, you’re most likely to work across multiple different K8s clusters: deploying your code to the dev cluster, trying out the latest feature on the integration cluster, doing a demo on the production cluster, and so on with your other clusters: CICD AWS, CICD VMware, Dev2, etc. Sometimes even at the same time on different windows and terminals. In this blog, I’ll share my workflow for switching between Kubernetes clusters, and how I juggle all the various Kubeconfig files for each cluster.
Kubeconfig, $KUBECONFIG, wha?
Just a quick primer on the various options for switching between different clusters, also sometimes referred to as contexts. As you already know, kubectl, our CLI interface to the world of Kubernetes, works off of a configuration file called kubeconfig which contains access information for one or more clusters.
By default, the kubectl binary looks for the kubeconfig file in this location
Overriding the default location of the config file can easily be done by passing in different kubeconfig files via the
command-line option or exporting a KUBECONFIG environment variable.
Even though the Kubeconfig yaml descriptor allows you to contain multiple cluster definitions in the same file, in practice there is nontrivial overhead to adding and removing lines from different parts of a growing file. And what do you do when your friendly colleague asks for access to a specific cluster? Now I have to surgically extract the clusters, user, and context into a new file (or remember how to use kubectl config view --minify).
If you have a handful of static clusters and/or okay with adding all the entries to a single Kubeconfig file (the default ~/.kube/config or otherwise), then I highly recommend using an awesome tool like kubectx to quickly switch between different clusters and contexts.
Quick Switch: Kctx & Klctx
I really like the simplicity of having separate Kubeconfig files per cluster. Adding and/or removing cluster kubeconfig files is a simple copy or delete file operation.
The solution which has worked best for me is:
- Add cluster Kubeconfig files to the ~/.kube directory but keep a consistent name configuration. For example, here are the files in my current ~/.kube directory:
- The ~/.kube/config is a symlink - (has Windows caught up yet?) to one of the kubeconfig files in this directory.
Kubectl (and virtually all other K8s tools) by default use ~/.kube/config, so all new terminals will point to the Kubernetes cluster the symlink resolves to. Now, using basic shell commands you can switch the currently selected cluster - e.g:
Even though that’s not too bad - I made a tiny helper script to help switch the context by changing the ~/.kube/config to the selected cluster’s kubeconfig. Whenever I run kctx (yes, borrowed the name), I get an interactive CLI tool to pick the cluster I’d like to switch to.
Like most engineers, I do work with multiple terminals (tmux is my weapon of choice). Most of my terminals point to my dev cluster, but a handful of terminals point to the CICD cluster and others. To switch the current terminal’s cluster, use the handy klctx tool (the “L” stands for local). The tool works exactly like kctx, except it sets KUBECONFIG to the designated cluster.
You can download the latest copy of these tools here: kctx-and-klctx. Just add the snippet to your shell startup script (e.g: $HOME/.bashrc). There is a dependency to fzf, so make sure to install it as well!
Which cluster?
How to visualize which cluster you’re targeting? I prefer seeing the cluster name in the shell prompt, like in the screenshot below you can clearly tell my terminal is targeting the spectro-dev cluster:
There are some really comprehensive solutions like kube-ps1 which do a fantastic job of providing a myriad of options to configure the look-n-feel of the shell prompt for the currently selected Kubernetes. Please check out the project, it’s fantastic!
In my workflow, most of my terminals target the default cluster, the cluster pointed to by ~/.kube/config symlink. So instead of showing the name of the cluster in all terminal prompts, I prefer seeing the cluster name in the tmux status-bar. For the few exceptional cases where a specific terminal overrides KUBECONFIG (using klctx or otherwise), the name of the cluster is shown in the shell prompt.
Anyway, this is my typical workflow for switching between various Kubernetes clusters and contexts. Do you like it? Hate it? Or if you have other suggestions for making it better and any questions, please drop me a tweet!
