Our Expertise

Ways to host a MongoDB cluster on Kubernetes

Aashiq Jacob
Posted by Aashiq Jacob on April 27, 2022

MongoDB is one of the most used database programs amongst developers. It is an open-source, general purpose, document-based, distributed NoSQL database server that is especially popular with JavaScript projects. It is instrumental in managing vast databases. In this blog, I will explain two ways of hosting a MongoDB cluster on Kubernetes.

Containerization provides developers flexibility, versatility, and support for many deployment environments. MongoDB helps Kubernetes in the automation of various critical aspects within containerized applications. Now, there are multiple ways to host a MongoDB cluster on K8s. However, I will discuss two of the easiest options:

  • Using Community Kubernetes Operator.
  • Using a custom Docker Image and Deployments.

Let's see how you can use these options in detail.

1. Using Community Kubernetes Operator

Kubernetes Operator provides an interface to manage third-party applications just like Kubernetes-native objects. MongoDB Kubernetes Operator helps in creating, configuring, and managing MongoDB StatefulSet. The MongoDB Operators are of two types viz., MongoDB Community Operator and MongoDB Enterprise Kubernetes Operator. Both possess different sets of features and requirements. Let’s look at the steps to set up a cluster using the Community Kubernetes Operator.

mongodb1

    • Install the Community Kubernetes Operator
      • Clone this repository:

        git clone https://github.com/mongodb/mongodb-kubernetes-operator.git 
      • Run the following command to create cluster-wide roles and role-bindings in the default namespace:

        kubectl apply -f deploy/clusterwide 
      • For each namespace that you want the Operator to watch, run the following commands to deploy a Role, RoleBinding, and ServiceAccount in that namespace:

        kubectl apply -k config/rbac --namespace 
      • Install the Custom Resource Definitions.

        • - Invoke the following command:

          kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml 
        • - Verify that the Custom Resource Definitions installed successfully:

          kubectl get crd/mongodbcommunity.mongodbcommunity.mongodb.com 
      • Install the Operator.

        • - Invoke the following kubectl command to install the Operator in the specified namespace:

          kubectl create -f config/manager/manager.yaml --namespace <my-namespace>  
        • - Verify that the Operator installed successfully:

          kubectl get pods --namespace <my-namespace> 
    • Generate Self-signed CA Certificate

      • Generate the CA key

        openssl genrsa -out rootca.key 4096 
      • Configure rootca.cnf

        # For the CA policy 
        [ policy_match ] 
        countryName = match 
        stateOrProvinceName = match 
        organizationName = match 
        organizationalUnitName = optional 
        commonName = supplied 
        emailAddress = optional 
        [ req ] default_bits = 4096 default_keyfile = rootca.pem ## The default private key file name. default_md = sha256 ## Use SHA-256 for Signatures distinguished_name = req_dn req_extensions = v3_req x509_extensions = v3_ca # The extentions to add to the self-signed cert
        [ v3_req ] subjectKeyIdentifier = hash basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment nsComment = "OpenSSL Generated Certificate for TESTING only. NOT FOR PRODUCTION USE." extendedKeyUsage = serverAuth, clientAuth
        [ req_dn ] countryName = Country Name (2 letter code) countryName_default = IN
        countryName_min = 2 countryName_max = 2
        stateOrProvinceName = State or Province Name (full name) stateOrProvinceName_default = Pune stateOrProvinceName_max = 64
        localityName = Locality Name (eg, city) localityName_default = Pune localityName_max = 64
        organizationName = Organization Name (eg, company) organizationName_default = TestComp organizationName_max = 64
        organizationalUnitName = Organizational Unit Name (eg, section) organizationalUnitName_default = TestComp organizationalUnitName_max = 64
        commonName = Common Name (eg, YOUR name) commonName_max = 64
        [ v3_ca ] # Extensions for a typical CA
        subjectKeyIdentifier=hash basicConstraints = critical,CA:true authorityKeyIdentifier=keyid:always,issuer:always
      • Generate CA Certificate

        openssl req -new -x509 -days 36500 -key rootca.key -out rootca.crt -config rootca.cnf 
      • Upload CA Certificate to Kubernetes Cluster

        kubectl create configmap ca-config-map --from-file=”~/ca.crt” --namespace <your-namespace> 
        kubectl create secret tls ca-key-pair  --cert=”~/ca.crt”  --key=”~/ca.key” --namespace <your-namespace> 
    • Create a MongoDB replica set in Kubernetes

      • Install Cert Manager.

      • Create the Cert Manager issuer, this will create the certificates required by the MongoDB replica set.

        cat <<EOF | kubectl apply -f - 
        apiVersion: cert-manager.io/v1 
        kind: Issuer 
        metadata: 
        name: ca-issuer-mongo 
          namespace: mongodb 
        spec: 
          ca: 
            secretName: ca-key-pair 
        --- 
        apiVersion: cert-manager.io/v1 
        kind: Certificate 
        metadata: 
          name: cert-manager-certificate 
          namespace: mongodb 
        spec: 
          secretName: mongodb-tls 
          issuerRef: 
            name: ca-issuer-mongo 
            kind: Issuer 
          commonName: "*.mongo-replicaset-svc.mongodb.svc.cluster.local" 
          dnsNames: 
          - "*.mongo-replicaset-svc.mongodb.svc.cluster.local" 
          - mongo-replicaset-0.com 
          - mongo-replicaset-1.com 
          - mongo-replicaset-2.com 
        EOF 
      • Use the Operator to create the MongoDB Replica set.

        cat <<EOF | kubectl apply -f - 
        apiVersion: mongodbcommunity.mongodb.com/v1 
        kind: MongoDBCommunity 
        metadata: 
          name: mongo-replicaset 
          namespace: mongodb 
        spec: 
          members: 3 
          type: ReplicaSet 
          version: "5.0.2" 
          replicaSetHorizons: 
          - horizon: mongo-replicaset-0.com:27017 
          - horizon: mongo-replicaset-1.com:27017 
          - horizon: mongo-replicaset-2.com:27017 
          security: 
            tls: 
              enabled: true 
              certificateKeySecretRef: 
                name: mongodb-tls 
              caConfigMapRef: 
                name: ca-config-map 
        
            authentication: 
              modes: ["SCRAM"] 
          users: 
            - name: admin 
              db: admin 
              passwordSecretRef: # a reference to the secret that will be used to generate the user's password 
                name: admin-password 
              roles: 
                - name: clusterAdmin 
                  db: admin 
                - name: userAdminAnyDatabase 
                  db: admin 
                - name: root 
                  db: admin 
              scramCredentialsSecretName: admin-scram 
            - name: dumpUser 
              db: admin 
              passwordSecretRef: # a reference to the secret that will be used to generate the user's password 
                name: dumpuser-password 
              roles: 
                - name: readWriteAnyDatabase 
                  db: admin 
              scramCredentialsSecretName: dumpuser-scram 
          additionalMongodConfig: 
            storage.wiredTiger.engineConfig.journalCompressor: zlib 
          statefulSet: 
            spec: 
              volumeClaimTemplates: 
                - metadata: 
                    name: data-volume 
                  spec: 
                    storageClassName: mongodb-ssd-storage 
                    accessModes: [ "ReadWriteOnce" ] 
                    resources: 
                      requests: 
                        storage: 512Gi 
        --- 
        apiVersion: v1 
        kind: Secret 
        metadata: 
          name: admin-password 
          namespace: mongodb 
        type: Opaque 
        stringData: 
          password: password 
        --- 
        apiVersion: v1 
        kind: Secret 
        metadata: 
          name: dumpuser-password 
          namespace: mongodb 
        type: Opaque 
        stringData: 
          password: password 
        EOF 
      • Expose the MongoDB replica set to the internet by creating Load Balancer service.

        cat <<EOF | kubectl apply -f - 
        apiVersion: v1 
        kind: Service 
        metadata: 
          name: mongo-replicaset-0 
          namespace: mongodb 
        spec: 
          ports: 
          - port: 27017 
            protocol: TCP 
            targetPort: 27017 
          selector: 
            statefulset.kubernetes.io/pod-name: mongo-replicaset-0 
          type: LoadBalancer 
        --- 
        apiVersion: v1 
        kind: Service 
        metadata: 
          name: mongo-replicaset-1 
          namespace: mongodb 
        spec: 
          ports: 
          - port: 27017 
            protocol: TCP 
            targetPort: 27017 
          selector: 
            statefulset.kubernetes.io/pod-name: mongo-replicaset-1 
          type: LoadBalancer 
        --- 
        apiVersion: v1 
        kind: Service 
        metadata: 
          name: mongo-replicaset-2 
          namespace: mongodb 
        spec: 
          ports: 
          - port: 27017 
            protocol: TCP 
            targetPort: 27017 
          selector: 
            statefulset.kubernetes.io/pod-name: mongo-replicaset-2 
          type: LoadBalancer 
        EOF 

The IP address generated from the Load balancer SVC should be bound to the domain names mentioned in the replicaSetHorizons, for example: mongo-replicaset-0.com, mongo-replicaset-1.com, mongo-replicaset-2.com.

2. Using a custom Docker Image and Deployments

While using an operator to deploy a MongoDB cluster sure makes the cluster creation simpler, you lose a lot of control over the MongoDB cluster setup. For instance, to expand the Persistent Volume (PV) size, you won't be able to do it without downtime. You need to use self-signed certificates since no CA provides certificates for the local domain.

To tackle the challenges of controlling the MongoDB cluster in Kubernetes, you can use the official MongoDB Docker image with some modifications to make it production-ready. The official MongoDB Docker image out of the box doesn't have authentication enabled. It is pretty trickly to enable it after creating the container. You can use the official MongoDB image to create a custom image with authentication, log rotation, and Replica set initiated.

mongodb2

    • Create Custom MongoDB image.

      Using the Dockerfile and MongoDB scripts makes it easy to create an Admin, a Database, and a Database User when the container is first launched. Create the following files in the exact location as mentioned in the header of the code snippet.

      • ~/MongoDB/Dockerfile

        FROM mongo:5.0.6 
        ENV DEBIAN_FRONTEND noninteractive ENV DEBCONF_NONINTERACTIVE_SEEN true
        RUN apt-get -qqy update \ && apt-get -qqy upgrade \ && apt-get -qqy install -y netcat \ && apt-get -qqy install rsyslog
        # Add scripts ADD scripts /scripts RUN chmod +x /scripts/*.sh RUN touch /.firstrun
        RUN (crontab -l ; echo "0 0 * * * bash /scripts/logs.sh") | crontab
        # Command to run ENTRYPOINT ["/scripts/run.sh"]
        # Expose listen port EXPOSE 27017 EXPOSE 28017
        # Expose our data volumes VOLUME ["/data"]
      • ~/MongoDB/scripts/first_run.sh

        #!/bin/bash 
        USER=$MONGODB_USERNAME 
        PASS=$MONGODB_PASSWORD 
        DB=$MONGODB_DBNAME 
        ROLE=$MONGODB_ROLE 
        PRIMARY_HOST=$HOST 
        # Start MongoDB service /usr/bin/mongod --dbpath /data --nojournal & while ! nc -vz localhost 27017; do sleep 1; done
        # Create User only on the Primary Pod. echo "Creating user: \"$USER\"..."
        a="$HOSTNAME" b=${a%-*-*}
        if [ ! -f /data/admin-user.lock ]; then sleep 60; touch /data/admin-user.lock if [ "$b" = "$PRIMARY_HOST" ]; then mongo $DB --eval "db.createUser({ user: '$USER', pwd: '$PASS', roles: [ { role: '$ROLE', db: '$DB' } ] });" fi;
        /usr/bin/mongod --dbpath /data --shutdown fi;
        /usr/bin/mongod --dbpath /data --shutdown echo "========================================================================" echo "MongoDB User: \"$USER\"" echo "MongoDB Database: \"$DB\"" echo "MongoDB Role: \"$ROLE\"" echo "========================================================================"
        rm -f /.firstrun
      • ~/MongoDB/scripts/run.sh

        #!/bin/bash 
        set -e 
        # Initialize first run if [[ -e /.firstrun ]]; then /scripts/first_run.sh fi
        # Startup cron for log rotation. cron
        echo "Starting MongoDB..."
        /usr/bin/mongod --replSet $REPLICA_ID --dbpath /data --bind_ip 0.0.0.0 --clusterAuthMode keyFile --keyFile /etc/secrets-volume/mongodb-keyfile --setParameter authenticationMechanisms=SCRAM-SHA-256 --auth --logpath /data/mongodb.log;
        exec "$@"
      • ~/MongoDB/scripts/logs.sh

        #!/bin/sh 
        # Log directory LOGDIR=/log-volume
        # Maximum number of archive logs to keep MAXNUM=5
        #Log files to be handled in that log directory files=(mongodb.log)
        for LOGFILE in "${files[@]}" do
        ## Check if the last log archive exists and delete it. if [ -f $LOGDIR/$LOGFILE.$MAXNUM.gz ]; then rm $LOGDIR/$LOGFILE.$MAXNUM.gz fi
        NUM=$(($MAXNUM - 1))
        ## Check the previous log file. while [ $NUM -ge 0 ] do NUM1=$(($NUM + 1)) if [ -f $LOGDIR/$LOGFILE.$NUM.gz ]; then mv $LOGDIR/$LOGFILE.$NUM.gz $LOGDIR/$LOGFILE.$NUM1.gz fi
        NUM=$(($NUM - 1)) done
        # Compress and clear the log file if [ -f $LOGDIR/$LOGFILE ]; then cat $LOGDIR/$LOGFILE | gzip > $LOGDIR/$LOGFILE.0.gz cat /dev/null > $LOGDIR/$LOGFILE fi
        done

Run Docker "build" in the MongoDB folder. Once built, you can tag and push the image to the container registry of your choice.

  • Deploy a MongoDB Cluster using Deployments.

    Let's create a three-node MongoDB cluster with one primary and two secondary nodes, which will be three different deployments in Kubernetes.

        • Create a Keyfile secret for the MongoDB cluster to communicate among the nodes. Make sure to base64 encode the key and replace "CHANGEME".

          cat <<EOF | kubectl apply -f - 
          apiVersion: v1 
          data: 
            mongodb-keyfile: CHANGEME 
          kind: Secret 
          metadata: 
            name: mongo-key 
            namespace: mongo-db 
          type: Opaque 
          EOF 
        • Create PVC which would create a PV for each deployment.

          cat <<EOF | kubectl apply -f - 
          --- 
          apiVersion: v1 
          kind: PersistentVolumeClaim 
          metadata: 
            name: mongo-0-disk 
            namespace: mongo-db 
          spec: 
            accessModes: 
            - ReadWriteOnce 
            resources: 
              requests: 
                storage: 512Gi 
          --- 
          apiVersion: v1 
          kind: PersistentVolumeClaim 
          metadata: 
            name: mongo-1-disk 
            namespace: mongo-db 
          spec: 
            accessModes: 
            - ReadWriteOnce 
            resources: 
              requests: 
                storage: 512Gi 
          --- 
          apiVersion: v1 
          kind: PersistentVolumeClaim 
          metadata: 
            name: mongo-2-disk 
            namespace: mongo-db 
          spec: 
            accessModes: 
            - ReadWriteOnce 
            resources: 
              requests: 
                storage: 512Gi 
          --- 
          EOF 
        • Create three different deployments which would act as three different Mongo nodes.

          cat <<EOF | kubectl apply -f - 
          apiVersion: apps/v1 
          kind: Deployment 
          metadata: 
            namespace: mongo-db 
            name: mongodb0 
          spec: 
            replicas: 1 
            selector: 
              matchLabels: 
                app: mongo-0 
            template: 
              metadata: 
                labels: 
                  app: mongo-0 
              spec: 
                terminationGracePeriodSeconds: 10 
                containers: 
                  - name: mongo 
                    image: containerregistry.azurecr.io/mongodb:5.0.6 
                    env: 
                      - name: MONGODB_USERNAME 
                        value: admin 
                      - name: MONGODB_PASSWORD 
                        value: password 
                      - name: MONGODB_DBNAME 
                        value: admin 
                      - name: MONGODB_ROLE 
                        value: root 
                      - name: REPLICA_ID 
                        value: mongoRS 
                      - name: HOST 
                        value: mongodb0 
                    ports: 
                      - containerPort: 27017 
                    volumeMounts: 
                      - name: mongo-key 
                        mountPath: "/etc/secrets-volume" 
                        readOnly: true 
                      - name: mongo-persistent-storage 
                        mountPath: /data 
                volumes: 
                - name: mongo-key 
                  secret: 
                    defaultMode: 0400 
                    secretName: mongo-key 
                - name: mongo-persistent-storage 
                  persistentVolumeClaim: 
                    claimName: mongo-0-disk 
          --- 
          apiVersion: apps/v1 
          kind: Deployment 
          metadata: 
            namespace: mongo-db 
            name: mongodb1 
          spec: 
            replicas: 1 
            selector: 
              matchLabels: 
                app: mongo-1 
            template: 
              metadata: 
                labels: 
                  app: mongo-1 
              spec: 
                terminationGracePeriodSeconds: 10 
                containers: 
                  - name: mongo 
                    image: containerregistry.azurecr.io/mongodb:5.0.6 
                    env: 
                      - name: MONGODB_USERNAME 
                        value: admin 
                      - name: MONGODB_PASSWORD 
                        value: password 
                      - name: MONGODB_DBNAME 
                        value: admin 
                      - name: MONGODB_ROLE 
                        value: root 
                      - name: REPLICA_ID 
                        value: mongoRS 
                      - name: HOST 
                        value: mongodb0 
                    ports: 
                      - containerPort: 27017 
                    volumeMounts: 
                      - name: mongo-key 
                        mountPath: "/etc/secrets-volume" 
                        readOnly: true 
                      - name: mongo-persistent-storage 
                        mountPath: /data 
                volumes: 
                - name: mongo-key 
                  secret: 
                    defaultMode: 0400 
                    secretName: mongo-key 
                - name: mongo-persistent-storage 
                  persistentVolumeClaim: 
                    claimName: mongo-1-disk 
          --- 
          apiVersion: apps/v1 
          kind: Deployment 
          metadata: 
            namespace: mongo-db 
            name: mongodb2 
          spec: 
            replicas: 1 
            selector: 
              matchLabels: 
                app: mongo-2 
            template: 
              metadata: 
                labels: 
                  app: mongo-2 
              spec: 
                terminationGracePeriodSeconds: 10 
                containers: 
                  - name: mongo 
                    image: containerregistry.azurecr.io/mongodb:5.0.6 
                    env: 
                      - name: MONGODB_USERNAME 
                        value: admin 
                      - name: MONGODB_PASSWORD 
                        value: password 
                      - name: MONGODB_DBNAME 
                        value: admin 
                      - name: MONGODB_ROLE 
                        value: root 
                      - name: REPLICA_ID 
                        value: mongoRS 
                      - name: HOST 
                        value: mongodb0 
                    ports: 
                      - containerPort: 27017 
                    volumeMounts: 
                      - name: mongo-key 
                        mountPath: "/etc/secrets-volume" 
                        readOnly: true 
                      - name: mongo-persistent-storage 
                        mountPath: /data 
                volumes: 
                - name: mongo-key 
                  secret: 
                    defaultMode: 0400 
                    secretName: mongo-key 
                - name: mongo-persistent-storage 
                  persistentVolumeClaim: 
                    claimName: mongo-2-disk 
          --- 
          EOF 
        • Create a service of type Load balancer to expose the MongoDB pods.

          cat <<EOF | kubectl apply -f - 
          apiVersion: v1 
          kind: Service 
          metadata: 
            namespace: mongo-db 
            name: mongo-lb-0 
          spec: 
            type: LoadBalancer 
            ports: 
            - protocol: TCP 
              port: 27017 
              targetPort: 27017 
            selector: 
              app: mongo-0 
          --- 
          apiVersion: v1 
          kind: Service 
          metadata: 
            namespace: mongo-db 
            name: mongo-lb-1 
          spec: 
            type: LoadBalancer 
            ports: 
            - protocol: TCP 
              port: 27017 
              targetPort: 27017 
            selector: 
              app: mongo-1 
          --- 
          apiVersion: v1 
          kind: Service 
          metadata: 
            namespace: mongo-db 
            name: mongo-lb-2 
          spec: 
            type: LoadBalancer 
            ports: 
            - protocol: TCP 
              port: 27017 
              targetPort: 27017 
            selector: 
              app: mongo-2 
          --- 
          EOF 

    After a while, the service mongo-lb-0, mongo-lb-1, and mongo-lb-2 will be assigned a public IP. Then you can bind a domain name to the IP (optional). Suppose the domain names are mongo-DB-0.com, mongo-DB-1.com, and mongo-DB-2.com.

      • Once all the resources are created, access the primary pod through a bash shell.

        kubectl -n mongo-db exec -it mongodb0-68fb678849-tb558 -- bash
      • Create a Replica set after executing the Mongo shell within the primary pod. Use the Domain name or the Load balancer IP created while creating the SVC.

        $ mongo 
        > db.getSiblingDB("admin").auth("admin", "password") 
        1 
        >rs.initiate({ _id: "mongoRS", version: 1,  
          members: [  
          { _id: 0, host: "mongo-db-0.com:27017" },  
          { _id: 1, host: "mongo-db-1.com:27017" },  
          { _id: 2, host: "mongo-db-2.com:27017" } ]}); 
        mongoRS:PRIMARY> 

        Now, you can connect to the Replica set using the database's connection string.

        mongosh
        "mongodb://admin:password@10.0.0.1:27017,10.0.0.2:27017,10.0.0.3:27017/admin?authSource=admin&replicaSet=mongoRS"

    Even though using separate deployments for each node of the MongoDB Replica set is a time-consuming task, you can have complete control over the Replica set. Also, in the case of PV expansion, you can increase the PV size of each Secondary Node individually. All this with zero time while performing any operation on MongoDB.

    You can efficiently run a highly available MongoDB cluster on Kubernetes using any of the above methods. While each technique has its pros, it is up to the business use case on which cluster type is required. To sum it all up, you can quickly create and deploy the cluster using the Kubernetes Operator to deploy MongoDB. But you will lose many of the customization features. On the other hand, using custom Docker images and individual deployments is more complex than using the Operator. But you will get complete control over the database configurations and customization options for the configurations as per the use case. The choice is yours!

Topics: Kubernetes, MongoDB, MongoDB cluster

Leave Comment

Subscribe Email

    Post By Topic

    See all