What is Multus CNI?
Multus CNI is an open-source Container Network Interface. Unlike other popular Kubernetes CNI projects, such as Cilium, Calico and Flannel, Multus is designed natively to provide advanced Kubernetes networking capabilities, particularly multi-network support.
That means it enables your Kubernetes pods to connect to multiple networks by integrating multiple CNIs into a single Kubernetes cluster.
Multus’s multi-network support is particularly relevant for scenarios such as NFV (Network Function Virtualization) use cases or complex multi-tenant environments.
At Spectro Cloud, we use Multus in our Virtual Machine Orchestration (VMO) capability, and you can add it to your clusters (or any of the other CNIs mentioned above) through your Palette Cluster Profiles.
Why do you need to remove Multus from a cluster?
Most guides and readmes focus on how to install a piece of software — so it’s easy enough to deploy Multus! But there are several reasons why you might need to uninstall CNI like Multus from your clusters.
You might be:
- Switching CNIs: Adopting a different CNI that better fits your evolving networking requirements.
- Decommissioning a cluster: Preparing a cluster for a new use case that doesn't require the current CNI.
- Troubleshooting: Addressing persistent network issues by removing and reinstalling the CNI to achieve a clean state.
In our use case, we needed a reliable way to uninstall Multus, without destroying our current cluster.
Uninstalling Kubernetes CNIs is not easy!
Kubernetes doesn't provide a standard method for uninstalling CNIs, leaving users to manage the uninstallation process themselves if they ever choose to switch CNIs.
CNIs integrate deeply with container runtimes, so removing them can be complex. Unlike installing a CNI, which is well-documented and supported, uninstalling often lacks clear guidelines.
This lack of a standardized uninstallation process can be particularly troublesome for complex CNIs like Multus, which integrates multiple network interfaces into pods.
Entering a standard `kubectl delete multus-daemonset.yaml` from the command line will leave residual config files dangling that’ll leave your cluster in a broken state, making pod creation impossible.
Here's what typically happens:
Errors on new pods
After deleting the Multus with `kubectl delete -f multus-daemonset.yml`, and then attempting to apply any new pod, you’ll encounter failures like this on the new pod:
This error indicates that Kubernetes is still trying to use Multus configurations that should have been removed. Once this happens, the new pod will never end up actually starting up successfully and will be stuck with the ContainerCreating status.
![This error indicates that Kubernetes is still trying to use Multus configurations](https://cdn.prod.website-files.com/64196dbe03e13c204de1b1c8/6682959d84d5f60048e493aa_AD_4nXfS--LmrPz2GV_RBhywIP0KfYBeQ6RWbzejJEPB_kL3BHoEvQ97Wxmq0GWkeQEYsypL47E6sknkR6LDW5helLRej-uSlDyxT9MriE_thuzz__i9aic-chr2bQb9P5tshcKurpPhekFP367oeKNkvS-T6Gwb.gif)
Dangling files are to blame
The primary culprits for this issue are the lingering configuration files and binaries that Multus installs, particularly:
- /etc/cni/net.d/00-multus.conf
- /opt/cni/bin/multus
These files need to be manually removed to fully uninstall Multus and restore normal pod operations.
Manual cleanup works, but it’s slow
The solution involves a thorough cleanup of these residual files. This can be done manually, but it's error-prone and time-consuming.
What if there was a way to automate this cleanup process?
It’s time to introduce spectro-cleanup.
Introducing spectro-cleanup
Spectro-cleanup is an open source tool developed by Spectro Cloud.
It provides an easy way to cleanup arbitrary files from nodes and resources from a K8s cluster.
It’s a general-purpose tool, but we can use it to target the cleanup of the dangling files left after Multus is uninstalled, ensuring our cluster is in a good state.
Spectro-cleanup can be run as either a DaemonSet or a Job, which provides the flexibility to perform cleanup operations without needing to install additional software on the nodes.
Here’s a brief overview of its components:
- ConfigMap: Defines which files and resources to clean up.
- DaemonSet/Job: Executes the cleanup based on the ConfigMap’s instructions.
- ServiceAccount and RBAC: Grants necessary permissions to allow spectro-cleanup to perform the cleanup.
Once the files we’ve chosen to delete are cleaned up, spectro-cleanup can then delete the configmaps used to define the files/resources to be deleted and finally delete the daemonset or job used to execute the cleanup.
Using spectro-cleanup to properly uninstall Multus
To uninstall Multus CNI properly using spectro-cleanup, follow these steps:
Step 1: Configure spectro-cleanup
Create a Kubernetes Job or DaemonSet to run spectro-cleanup. Below is the YAML configuration tailored for Multus cleanup via a Job:
Step 2: Run spectro-cleanup
Deploy the job using kubectl:
![run spectro-cleanup](https://cdn.prod.website-files.com/64196dbe03e13c204de1b1c8/6682959d42846f18521a381c_AD_4nXesdW9T6AMwBpDQqXtRanOMe3ez26hgetNDk9RZ234kkw4JJ2u9016IjixrenOI8YXmD1UVqG3t2CV_kQqqe3Bwp10y6y2J03MdqCXGLsI8inxL_EuTVZ3Gtd-GCDSxppU6b09l25WCzp_mK86ogzSFxyc.gif)
Note: This step NEEDS to be run prior to uninstalling Multus. If you already ran the command to delete Multus before this step, then spectro-cleanup will fail to start, due to the same errors that we discussed earlier in this blog post.
Step 3: Uninstall Multus
Proceed to uninstall Multus by deleting the DaemonSet:
Ensure that all Multus components are removed.
Step 4: Verify by Creating a New Pod
Create a test pod to verify that it can actually start:
Deploy the pod:
Check the pod status and logs to confirm it starts without errors.
![Check the pod status and logs to confirm it starts without errors.](https://cdn.prod.website-files.com/64196dbe03e13c204de1b1c8/6682959dacb238c6d9e22893_AD_4nXcDBqMukmxXRmOBQrXwmDKOIoDJXPr9UQ3dpR5l6K9wj-Fcd9S0b_ObCLKOGl1fdqjh13p39H9DdvMegDQv5OWfRcXOq-OC73O3m3h4I35awxZgPKCqZ6lPA567ogq1QtGeAUqV_-Z8GqJvwIqy62xyntqi.gif)
From cleanup to Validator
Spectro-cleanup is a standalone tool, but it’s also integrated into our validator ecosystem. The Validator ecosystem, including plugins like OCI, Network, and AWS, helps validate rules against your environment and potentially alert on any validation failures.
- Validator Plugins: Validates user configured rules against specific cloud or infrastructure components (e.g., OCI, AWS), and then generate ValidationResult custom resources
- Validator: Manages installation of validator-plugins and consumes ValidationResults, publishing them to the sink of your choice (e.g. Slack, Alertmanager)
Spectro-cleanup is used by the validator to automate the removal of all plugin and validator resources when validator is uninstalled.
Wrapping up
Uninstalling Multus CNI without proper cleanup can cause significant disruptions to your Kubernetes cluster.
By using spectro-cleanup, you can automate the removal of residual files, ensuring a smooth and error-free uninstallation process.
Follow the steps outlined in this guide to maintain a clean and functional Kubernetes environment.
For more details on spectro-cleanup, visit the spectro-cleanup github repo. Whether you’re managing a small cluster or a large-scale deployment, spectro-cleanup offers a reliable solution for cleaning up any unwanted files or K8s resources.