Multitenancy is great… but it has side effects
As Kubernetes adoption grows in large organizations, so does the challenge of managing resource usage across a growing number of tenants. A key concern in Kubernetes multi tenancy is the "noisy neighbor" problem. This is where workloads unintentionally disrupt others by overconsuming shared cluster resources, such as CPU and memory, causing issues with application quality of service (QoS).
The word “tenant” in multi-tenancy can refer to many things. Tenants might be individual developers on a single team, teams or business units within an enterprise, or even multiple businesses.
Multi-tenancy is therefore often divided into “soft” and “hard”, based on level of trust between tenants. In a soft multi-tenant environment, tenants can be expected to trust each other because they’re working towards the same goals as part of the same organization. However, they can still unintentionally disrupt the workloads of other tenants.
In this post, we’ll explore how cluster administrators can address noisy neighbor issues between tenants in soft multi-tenant environments, keeping workloads working optimally and meeting the organization’s needs.
Isolating workloads to reduce resource contention
The first step in addressing noisy neighbors is establishing clear boundaries between tenants, to prevent resource contention.
This can be challenging when workloads must communicate with each other. More options are available to us in ‘hard’ multitenancy situations, where workloads operate independently and there is no limit to how much they can be separated from each other.
Physical isolation with single-tenant clusters
The strongest form of workload isolation you can have in Kubernetes is to run fully separate clusters per tenant.
When workloads are placed on different clusters, they do not share worker nodes, preventing resource contention at the worker node level. Workloads are also then managed through multiple control planes.
This strategy is particularly useful in large organizations where teams or departments operate independently, and cooperation or coordination between them is minimal.
However, the many-clusters approach comes with tradeoffs:
- Each additional cluster means an additional control planemore control planes, and therefore more machines dedicated to managing workloads instead of running them.
- Fully segregating clusters with dedicated resources means lower utilization.
- Managing manylots of clusters can be challenging without effective automation (and this is where Cluster API comes in)
Logical isolation with namespaces
To address the drawbacks of isolating workloads physically by hosting them on different clusters, you can use logical isolation instead, using Kubernetes namespaces, and various tools that integrate with namespaces, such as Resource Quotas and RBAC.
Resource Quotas limit the total amount of CPU, memory, and storage that workloads can use within the namespace. This ensures that one tenant cannot overuse resources to the detriment of other tenants. Care must be taken to ensure each tenant’s quota meets their needs.
RBAC ties in with resource quotas because you want to ensure tenants can only deploy workloads to their own target namespace.
It’s about more than just workloads being deployed, too. It’s also about what workloads can do to other Kubernetes objects after being deployed.
Custom controllers can potentially act upon any Kubernetes resource using the Kube API. Imagine one ambitious tenant deploying a custom controller that looks for workloads it believes are misconfigured and automatically reconfigures them. The controller will be using a Kubernetes service account, and RBAC can restrict what operations that service account can perform. At the very least, RBAC can restrict that service account to only performing operations against Kubernetes objects in the same namespace, safeguarding other tenants’ workloads from the controller’s effects.
Resource management: allocating and controlling resource usage
After establishing boundaries to isolate workloads, the next step in mitigating noisy neighbors is effectively managing cluster resources.
Workloads should have access to the resources they need, but no single workload should monopolize shared cluster resources.
Setting resource requests and limits
Kubernetes provides a powerful mechanism for managing resource allocation through resource requests and limits, which are configured under the resources field in workload specifications (e.g., Deployments).
These settings control both how workloads are scheduled and how they operate. In a multi-tenant environment, they act as safeguards to maintain a fair and predictable environment in shared clusters for all tenants.
Requests define the guaranteed amount of CPU and memory a pod requires to run. Kubernetes considers these values when scheduling the pod, ensuring it is only placed on a node that has enough available resources to meet the request. This prevents workloads from being scheduled onto nodes that are already too heavily utilized.
Limits define the maximum amount of CPU and memory a pod can use. While CPU limits allow workloads to burst and consume idle CPU, memory limits are enforced more strictly. If a pod exceeds its memory limit, Kubernetes will kill the pod to free up memory for other workloads. This ensures that memory-intensive workloads cannot disrupt other pods by consuming all available memory on a node.
Requests and limits are set as part of workload specs, which is the realm of developers. What about cluster administrators? Cluster administrators can enforce policies using mutating or validating admission webhooks. For example, webhooks can:
- Require all workloads to define resource requests and limits.
- Prevent workloads from setting resource limits that are significantly higher than their requests, which helps avoid scenarios where pods monopolize idle resources during bursts.
Kubernetes has many extension points. Webhooks are just one example. Cluster administrators can take advantage of these extension points to make Kubernetes work best for them and their organization — including preventing tenants from overusing resources!
Managing requests and limits with Vertical Pod Autoscaler (VPA)
Many Kubernetes users are already familiar with the Horizontal Pod Autoscaler (HPA), which scales the number of pods automatically as resource usage grows.
In contrast, Vertical Pod Autoscaler (VPA) focuses on optimizing resource requests and limits for individual pods based on observed usage patterns. It monitors how workloads use CPU and memory over time, providing recommendations or automatically adjusting configurations to align with actual usage.
This makes VPA particularly valuable during Day 2 operations, especially for new workloads with uncertain resource needs or workloads with unpredictable or evolving demands.
VPA operates in three modes:
- "Auto" Mode: Automatically adjusts resource requests and limits based on usage data, ideal for organizations comfortable with Kubernetes proactively reconfiguring workloads.
- "Off" Mode: Provides resource recommendations without making changes, useful for learning optimal resource settings without immediate adjustments.
- "Initial" Mode: Sets resource values when pods are created but does not adjust them for running pods.
During the early stages of Day 2 operations, VPA’s "Off" mode is an excellent starting point. Administrators can observe recommendations, manually tune workloads, and gain confidence before enabling "Auto" mode for proactive optimization.
Using VPA to configure workloads can help prevent them from becoming noisy neighbors. By fine-tuning both CPU and memory requests, VPA helps reduce resource contention and cluster instability:
- CPU: If a workload consistently exceeds its CPU request, VPA increases the request to ensure it has enough guaranteed CPU, preventing resource contention that negatively impacts other tenants sharing the cluster. On the other handConversely, if a workload over-requests CPU but doesn’t fully utilize it, VPA reduces the request, freeing up capacity for other tenants.
- Memory: For workloads exceeding their memory requests, VPA increases the request to stabilize the workload and reduce the likelihood of eviction (pods exceeding memory requests are prioritized for eviction). Evictions can cause unnecessary resource consumption during pod restarts and rescheduling, creating noise that disrupts other workloads on the same node. By right-sizing memory requests, VPA helps minimize disruption and noise to other tenants.
Scaling and optimization: preparing for efficient growth
As Kubernetes clusters evolve, scaling and optimization become essential Dday 2 operations tasks for ensuring sustained performance, fair resource allocation, and cost efficiency.
Cluster autoscaling: adding nodes automatically as they’re needed
Variable demand can cause workloads to become noisy neighbors. We can mitigate this through cluster autoscaling, which automatically adjusts the number of worker nodes in the cluster based on current needs.
When a tenant’s workload spikes, the workload can burst if it is allowed to do so, and new pods will be created that enter a pending state due to insufficient resources on existing nodes. The cluster autoscaler will react to this by adding new nodes to accommodate the demand. This ensures thatthe the workload does not disrupt other tenants’tenant’s workload does not disrupt other tenants’ workloads by monopolizing shared resources. Once the demand lowers, unused nodes are removed to save costs.
An open-source tool for implementing this capability is Cluster Autoscaler, which works by monitoring pending pods and provisioning nodes as necessary. You can define the minimum and maximum number of nodes for the cluster to ensure control over scaling behavior.
For clusters deployed on cloud platforms, autoscaling is often provided as a built-in feature you can enable during cluster creation, powered by tools like Cluster Autoscaler.
Autoscaling can be more challenging to implement in on-premises environments, particularly bare metal, because resource pools are less elastic. Tools like Cluster API and MAAS help bring cloud-like flexibility to on-prem deployments by enabling administrators to manage clusters and their infrastructure as scalable resource pools. This makes it easier to dynamically add capacity in response to tenant demand spikes, reducing resource contention and noisy neighbor issues.
Cluster overprovisioning: adding nodes even before they’re needed
Another proactive approach to managing resources is overprovisioning, which involves running more worker nodes than are strictly needed to meet current workload demands.
By maintaining a buffer of excess capacity, pods can be scheduled immediately when demand spikes, rather than entering a pending state while the cluster adjusts by bringingto bring new nodes online. In other words, you trade efficiency for performance.
One open-source tool that simplifies overprovisioning is cluster-overprovisioner. This tool creates placeholder pods with low-priority settings to simulate additional capacity. When actual workloads require the resources, these placeholder pods are evicted, freeing up capacity for critical workloads.
Overprovisioning is especially useful in environments with highly dynamic or unpredictable workloads, where responsiveness is a top priority. By proactively maintaining spare capacity, administrators can reduce the impact of noisy neighbors during demand surges, ensuring a more stable experience for all tenants.
Monitoring and optimization with Prometheus and Grafana
Monitoring can be tedious to set up, but it’s critical to see exactly what’s going on in the cluster over time. With good monitoring practices, you can see and measure the symptoms of noisy neighbors, like workloads being throttled.
Node-level metrics such as node_cpu_seconds_total help identify nodes that are consistently running at full capacity. If nodes are frequently maxed out, it may indicate that noisy neighbors are using all the CPU resources.
Pod-level metrics like container_memory_working_set_bytes provide insights into pod resource usage. Pods consistently operating near their limits may require adjustments to resource requests and limits. For example, increasing resource requests can reduce the frequency of bursts, creating a more predictable resource usage pattern and leaving room for other workloads to burst if they need to.
To see these noisy neighbor problems in action, tools like Prometheus and Grafana provide visibility into cluster and workload performance, enabling administrators to detect and address resource contention before it becomes a problem.
By integrating Prometheus with alerting systems, administrators can receive notifications about potential noisy neighbors, such as nodes experiencing high resource contention or pods approaching their limits. This enables timely interventions, such as scaling up the cluster, adjusting resource quotas, or modifying workload configurations.
Your next steps
The problem of noisy neighbors is not new, but in the highly dynamic and interconnected world of a Kubernetes cluster, it’s a common challenge. You have many tools in your toolbox to tackle it, ranging from fully isolating workloads on different clusters, to configuring autoscaling. The right blend of approaches will depend on your unique situation: your mix of workloads and their usage profiles; the vagaries of your infrastructure and its capacity; and your tolerance for overprovisioning costly resources.
But whichever mix of tactics you employ, a Kubernetes management platform is a great foundation to support you. A platform like Palette not only makes it easier to stand up and configure multiple clusters (if that’s your approach), but also to deploy observability tools, control user access, and enforce standardized policies and configurations. If this sounds good to you, why not book a demo to see it in action?