Managing multi-cluster Kubernetes RBAC using Workspaces

Expanding RBAC for a multi-cluster world

Kubernetes has long offered Role Based Access Control (RBAC) features to help you assign permissions and control access to your cluster by both individual users and deployed software. 

But, like many aspects of Kubernetes, it can be challenging to manage RBAC at scale. And with an increasing focus on security, regulatory requirements, and deploying multiple clusters to achieve reliability (deploying even to the edge), this is only getting more challenging as time goes by.

When Kubernetes resources are deployed to a wide change of clusters, it can be difficult to keep track of them to ensure they don’t remain deployed where they shouldn’t be. 

This problem is especially true for RBAC resources. If resources such as role bindings and roles are unwittingly left active, bad actors can use them to access data or damage the system.

There are myriad open source tools out there to help you manage Kubernetes configurations, like Terraform and CI/CD tools such as GitHub Actions and ArgoCD. But, when it comes to tooling, cluster administrators need all the help they can get to automate and centralize their cluster configuration.

Spectro Cloud's Palette offers a concept called Workspaces that can unlock an easier way to manage RBAC constructs. Workspaces are essentially a logical grouping of clusters or namespaces. They help you manage RBAC at scale by providing a way for you to centralize your RBAC configurations and deploy them to multiple clusters in repeatable ways.

Let's walk through an example so you can start to see how it works.

In this walkthrough, we’ll deploy multiple clusters and set up a Workspace to synchronize role bindings across the clusters.

Step 1: Creating the Cluster Profile

Palette declaratively models everything that goes into a Kubernetes cluster through something called a Cluster Profile. Each ‘layer’ in a profile can contain a software component from our repositories, Helm charts, Zarf packages, or arbitrary manifests that you provide. 

This means a Cluster Profile layer is a great place to put the RBAC roles and subjects you want deployed to your clusters. 

When you create a Palette Workspace after the clusters have been created, the Workspace will ensure the role bindings are created on the clusters, activating the roles so that they grant permissions to the subjects.

This unlocks many different use cases, for example:

• An auditing tool that looks for patterns in ConfigMaps across all namespaces. The application would require read-only access to permissions for all ConfigMaps across all namespaces.
• A monitoring tool that needs read access to specific metrics across all namespaces. This tool would require permissions to gather performance data from various resources like Pods, Nodes, and Services to ensure the health and performance of the applications running in the cluster.
• A CI/CD pipeline that needs write access to deploy applications to specific namespaces. This pipeline would require permissions to create, update, and delete resources such as Deployments, Services, and ConfigMaps in those namespaces to automate the deployment and update processes.

Here, we'll create a role and service account subject for the ConfigMap auditor application in the examples above. We’ll create them in a layer in a profile for an AWS IaaS cluster.

Add a layer to your Cluster Profile with the following manifests for the ClusterRole and ServiceAccount:

 
 ```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: read-configmaps
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
```
‍
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: auditor
  namespace: default

The Palette GUI will display the new layer in the stack after you save it. Here, we’ve named the layer “rbac”. The layer is now considered part of the Cluster Profile and will be available for use deploying any cluster from the profile.

Step 2: Deploying the clusters

To show the RBAC system in action across multiple clusters, we’ll use Palette to deploy three Kubernetes clusters, applying the Cluster Profile we just built. 

Use the Cluster Profile created in the previous step to deploy each cluster. Start from the “Profiles” page in the Palette GUI.

Select the profile from the list of profiles displayed.

Select “Deploy” and proceed through the cluster creation wizard. Note that a cluster deployed with Palette requires at least 4 vCPU in total for all control plane nodes.

Repeat this procedure two more times, giving each cluster a unique name, so that you have three clusters.

Here they are provisioning — they’ll come up healthy shortly.

Step 3: Creating the Workspace

When the clusters have finished being deployed, we can begin creating a Workspace. 

When creating the Workspace, select only the first two of the three clusters created. Leave the third cluster out of the Workspace.

By default, there are no role bindings defined. Select “Add New Binding” to add the role binding we need.

While creating the role binding, specify the `read-configmaps` ClusterRole as the role and the `auditor` ServiceAccount as the subject.

The interface will reflect the added role binding after this step.

For the namespace of the workspace, use the name “default” to assign the default namespace to the workspace.

Step 4: Verifying RBAC rules have been applied

 Checking the Workspace clusters

Let’s now check that the RBAC rules we defined in our manifest have been properly applied to the two clusters we assigned to the new Workspace.

First, get the kubeconfig credentials for one of the two clusters in the Workspace from “Admin Kubeconfig File” in the Palette GUI.

Use `kubectl` to verify that the ClusterRole and ServiceAccount specified in the previous steps exist in the cluster.

```bash
kubectl get clusterrole read-configmaps
```
‍
```
NAME       CREATED AT
read-configmaps 2024-07-26T22:31:25Z
```
‍
```bash
kubectl get serviceaccount auditor
```
‍
```
NAME   SECRETS AGE
auditor 0    149m
```

These resources were created by Palette while it created the clusters. After it installed the base Kubernetes stack, it installed the "rbac" layer in the Cluster Profile.

Use `kubectl` to verify that a ClusterRoleBinding associated with the role and the service account exists in the cluster.

 ```bash
kubectl get clusterrolebinding | head -1 \
  && kubectl get clusterrolebinding | grep read-configmaps
```
‍
```
NAME                 ROLE            AGE
spectro-on-demand-4965938436135743191 ClusterRole/read-configmaps 12m
```

Because Palette defined and created the binding for you, its name does not matter. In this case, Palette named the binding "spectro-on-demand-4965938436135743191". The binding is linked to the ClusterRole "read-configmaps".

 
 ```bash
kubectl get clusterrolebinding spectro-on-demand-4965938436135743191
```
‍
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2024-07-20T00:55:09Z"
  labels:
    spectrocloud.com/workspace: "true"
    spectrocloud.com/workspaceId: 669b0ad9d628d65664fb58c2
  name: spectro-on-demand-4965938436135743191
  resourceVersion: "57288"
  uid: 67135021-e3a5-433f-9a90-8db6950c738a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: read-configmaps
subjects:
- kind: ServiceAccount
  name: auditor
  namespace: default
```

The binding is linked to the ServiceAccount "auditor" in the default namespace. Therefore, it correctly binds the ClusterRole and ServiceAccount to extend read-only permissions to the ServiceAccount. 

Any application deployed to the default namespace could use this ServiceAccount to gain the permissions to read all ConfigMaps in the cluster.

Use `kubectl` to verify that the ServiceAccount has the ability to read ConfigMaps in a namespace called “test-namespace”.

```bash
kubectl auth can-i get configmaps \
  --namespace=test-namespace --as=system:serviceaccount:default:auditor
```
‍
```
yes
```

Note how the service account has permission to read the ConfigMaps even though they are in a different namespace than the service account.

To recap, Workspaces enabled you to implement this security policy in the clusters defined in the Workspace. It activated the roles defined in the layer of the Cluster Profile.

Checking the non-Workspace cluster

Let's verify that the permissions were not extended to the service account in the third cluster that we deliberately left out of the Workspace. 

Get the kubeconfig credentials for the third cluster created previously (the one we left out of the Workspace creation step).

If you use `kubectl` to verify which ClusterRoles and ServiceAccounts have been created in the cluster, you'll notice that this cluster has the same resources as the Workspace clusters. This is because it was created from the same Palette Cluster Profile as the Workspace clusters.

 
 ```bash
kubectl get clusterrole read-configmaps
```
‍
```
NAME       CREATED AT
read-configmaps 2024-07-26T22:33:37Z
```
‍
```bash
kubectl get serviceaccount auditor
```
‍
```
NAME   SECRETS AGE
auditor 0    167m
```

Use `kubectl` to verify which ClusterRoleBindings exist in the cluster.

```
kubectl get clusterrolebindings | grep read-configmaps | wc -l
```
‍
```
0
```

Unlike with the first cluster, in this cluster, there are no ClusterRoleBindings at all. There is no binding to extend permissions to the ServiceAccount in this cluster.

Use `kubectl` to test whether the service account would have the ability to read ConfigMaps in a "test-namespace" namespace.

```
kubectl auth can-i get configmaps \
  --namespace=test-namespace --as=system:serviceaccount:default:auditor
```
‍
```
no
```

Unlike with the first cluster, in this cluster, an application deployed with the ServiceAccount would not have permissions to read ConfigMaps across all namespaces in the cluster.

So not only did Workspaces enable you to implement this security policy in the clusters defined in the Workspace, it also enabled you to do this without extending the permissions to subjects in other clusters. 

Workspaces allows you to centralize which software and manifests to deploy to your clusters, via Cluster Profiles, while only activating the RBAC permissions defined in the roles on the desired clusters, via Workspaces.

Maintaining the Workspace

You can modify the Workspace by going to the Workspaces part of the Palette GUI and selecting the Workspace.

You can:

• Change the Workspace settings by adding clusters to it, removing clusters from it, changing which namespaces in the clusters the Workspace applies to, and adding new role bindings. 
• Work with Workspaces features not covered in this tutorial such as setting quota, scheduling backups and restricting which container images may be deployed.
• View every workload deployed to the workspace in one place, whichever namespace and cluster it was deployed to.
• Assign clusters to multiple Workspaces, so you can define Workspaces for different business needs.

Here, the workload view shows an example deployment (taken from the Kubernetes website) deployed to the first cluster:

Level up your RBAC with Palette

Access controls govern how different job functions and teams can use your Kubernetes clusters and the resources inside them. But you also need to manage how different user accounts are granted access to the capabilities within the Palette management platform, too. 

Palette offers powerful granular RBAC controls that allow you to use similar RBAC techniques to control access to Palette features. For example, you can limit which users are able to create or modify Cluster Profiles or build new clusters. It’s all part of our commitment to helping you operate Kubernetes at scale, with the right security and governance in place.

If you’ve been wrestling with RBAC across a growing Kubernetes estate, the Workspaces concept we’ve outlined in this blog might be able to help. The best way to get started is by booking a demo. We’ll give you a tour of the platform and hand you the keys so you can explore yourself. Alternatively, read more in our docs or ask any questions on our community Slack.