Using Kustomize to Kubernetes your way to MongoDB ReplicaSet

Kustomize your way to MongoDB ReplicaSet

A replica set in MongoDB is a group of mongod processes that maintain the same data set. Replica sets provide redundancy and high availability and are the basis for production deployments.

In this post we will set up a MongoDB replica set with the abilities to be a production-ready environment. We are going to use a Kubernetes cluster, using kustomize to set up the whole system.

There are some capabilities that a Kubernetes system provides us; the biggest one is to maintain HA. If any pod/statefulset goes down Kubernetes will try to bring it up again, maintaining the number of replicas in the system that are configured by the user. Hence, while setting up the MongoDB replica set we also want to use the capabilities of Kubernetes to maintain high availability.

We want a capability to autoscale MongoDB ReplicaSet with the number of replicas of MongoDB StatefulSet. In order to do so we will set up a sidecar that will keep on monitoring the pod (replicas). If any new MongoDB StatefulSet’s replica comes up the sidecar will add it to the MongoDB ReplicaSet, and if the MongoDB StatefulSet replica scales down it will remove it from the existing MongoDB ReplicaSet. The code for sidecar you can check out on GitHub.

In our example we want our MongoDB to be set up with security features, so we will run MongoDB in a secure mode. We will make the communication between the mongo nodes secure with a mongo key; additionally we will start mongo with the “auth” option and will configure an “admin” user. We will run some scripts to set up the system according to our application needs and those scripts run only one time in a lifespan of MongoDB (only at start), with the help of docker-entrypoint.sh provided by MongoDB's docker image. We will load all those scripts into the filesystem with the help of ConfigMaps. We will supply passwords via K8s Secrets. Moreover, we don’t want to maintain a local copy of the image. Instead, we will be using the mongo image from hub.docker.com so that we can easily update mongo to the latest version.

Requirements

1. A K8s cluster.
2. Kustomize Installed.
3. Internet Accessibility.

We will be using the docker image provided by MongoDB and a sidecar image built from mongodb-k8s-sidecar.

Let's get started

I am using a GKE cluster but you can run the same on your local Kubernetes cluster too.

The MongoDB docker image has the ability to configure some scripts to run only when the first time MongoDB is started. We will take benefit of this feature and will set a root user password and will create one more user just to demo that we can design the system in such a way that all those scripts run only once in their life span.

Then we will make communication between the nodes of the replica set secure with a mongo key. This too we will demo in a way where you don't need anything except a YAML file.

Create Scripts and Secrets

First, let's create a script that we want to run the first time. We will load the scripts using configMap.

 
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-init
data:
mongo-user.sh: |
  mongo admin -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
      use unatnahs_db
      db.createUser({user: "unatanhs", pwd: "${SECOND_USER_DB_PASSWORD}", roles: [
          { role: "readWrite", db: "unatnahs_db" }
      ]});
  EOF

All these env variables we will fill with the help of mongo secrets:

 
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: mongosecret
data:
  mongoRootPassword: c2hhbnRhbnViYW5zYWw=
  unatnahsDbPassword: Y2hhbmdldGhlc2VwYXNzd29yZHM=

“mongoRootPassword” will be the password for the admin user named “root” and “unatnahsDbPassword” will be the password for the “unatnahs” user mentioned in the mongo-user.sh script in configMap.

Now we will create a mongo-key which will help to secure inter-node communication:

 
apiVersion: v1
kind: ConfigMap
metadata:
  name: mongo-key

data:
  mongo.key: |
    ahaksdnqsakdqnajhvckqaafnxasxaxaxmaskdadadsasfsdsdfsf
    schcacnctcacncuadasdadadfbsasddfbadadwsioweewvaas
    dfasasakjsvnaa

Change Script Permissions

When the Mongo runs we need to set specific permissions for mongo-key. Since we will be loading everything into the filesystem of the mongo container with the help of ConfigMap we may face an issue for the permissions as the mongo pod will start with the “mongo” user, not the “root” user. We first have to change permissions of the scripts on the fly and then start Mongo. To do that we will temporarily load the configMap of mongo-key into a temp location and then we will copy to a more obvious location. The reason for doing this is that the Container loads configMap as a symlink in the filesystem and that will not allow us to change the user and permission of the script. So first we will copy the file to another location and then we will change the user and the permission of the file. To do so, we have another script.

 
apiVersion: v1
kind: ConfigMap
metadata:
  name: mongo-scripts
data:
  mongo-data-dir-permission.sh: |
    chown -R mongodb:mongodb ${MONGO_DATA_DIR}
    cp -r /var/lib/mongoKeyTemp /var/lib/mongoKey
    chown -R mongodb:mongodb /var/lib/mongoKey
    chmod 400 /var/lib/mongoKey/mongo.key
    chown -R mongodb:mongodb /var/lib/mongoKey/mongo.key
	

Create K8s Service

Now we will create a Service which should not load balance.

 
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  ports:
    - port: 27017
      targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
	

‍Create Service Accounts

Now we will create a service account and cluster role binding. Our sidecar needs permission to watch the pod. You can control the permission. We need just a watch and list permission but the following gives a lot more permissions on the system.

 
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: mongo-account
    namespace: mongodb-repl-system
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: mongo-role
  rules:
  - apiGroups: ["*"]
    resources: ["configmaps"]
    verbs: ["*"]
  - apiGroups: ["*"]
    resources: ["deployments"]
    verbs: ["list", "watch"]
  - apiGroups: ["*"]
    resources: ["services"]
    verbs: ["*"]
  - apiGroups: ["*"]
    resources: ["pods"]
    verbs: ["get","list", "watch"]
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: mongo_role_binding
  subjects:
  - kind: ServiceAccount
    name: mongo-account
    namespace: mongodb-repl-system
  roleRef:
    kind: ClusterRole
    name: mongo-role
    apiGroup: rbac.authorization.k8s.io
	

MongoDB StatefulSet

Now let's just go to the final state and create a StatefulSet with two containers - one of the actual mongo and the other one as the sidecar.

 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
spec:
  podManagementPolicy: Parallel
  replicas: 1
  selector:
    matchLabels:
      role: mongo
  serviceName: mongo
  template:
    metadata:
      labels:
        role: mongo
    spec:
      serviceAccountName: mongo-account
      terminationGracePeriodSeconds: 30
      containers:
        - image: mongo:4.2
          name: mongo
          command: ["/bin/sh", "-c"]
          args:
            [
              "/home/mongodb/mongo-data-dir-permission.sh && docker-entrypoint.sh mongod --replSet=rs0 --dbpath=/var/lib/mongodb --bind_ip=0.0.0.0 --keyFile=/var/lib/mongoKey/mongo.key",
            ]
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              value: root
            - name: MONGO_DATA_DIR
              value: /var/lib/mongodb
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongosecret
                  key: mongoRootPassword
            - name: SECOND_USER_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongosecret
                  key: unatnahsDbPassword
          ports:
            - containerPort: 27017
          volumeMounts:
            - mountPath: /var/lib/mongodb
              name: mongo-data
            - name: mongoinit
              mountPath: /docker-entrypoint-initdb.d
            - name: mongopost
              mountPath: /home/mongodb
            - name: mongokey
              mountPath: /var/lib/mongoKeyTemp
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar:latest
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo"
            - name: KUBE_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_MONGO_SERVICE_NAME
              value: mongo
            - name: MONGODB_USERNAME
              value: root
            - name: MONGODB_DATABASE
              value: admin
            - name: MONGODB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongosecret
                  key: mongoRootPassword
      volumes:
        - name: "mongoinit"
          configMap:
            name: "mongo-init"
            defaultMode: 0755
        - name: "mongopost"
          configMap:
            name: "mongo-scripts"
            defaultMode: 0755
        - name: "mongokey"
          configMap:
            name: "mongo-key"
            defaultMode: 0755

  volumeClaimTemplates:
    - metadata:
        name: mongo-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi

In this StatefulSet we have loaded all the ConfigMap as required on specific locations. The StatefulSet first will run the permission change script, then the actual mongo script with required arguments.

VOILA!!!

We have set up a complete mongo replica set! Now you can just increase or decrease the ReplicaSet count with a simple:

 
  kubectl scale --replicas=3 statefulset mongo -n mongodb-repl-system
	

Now you can connect to the mongo with the user mentioned above on:

 
    mongo-0.mongo, mongo-1.mongo,mongo-2.mongo
	

Port forward to access mongo on local:

 
     kubectl -n mongodb-repl-system port-forward svc/mongo 27017
	

Conclusion

We set up a secure MongoDB replica set on the K8s cluster using kustomize with the help of mongo’s own docker image.

PS: You can check out the code on GitHub. There either you can use a single manifest file or the properly segregated files for better readability.