Docker & Kubernetes : Cassandra with a StatefulSet

bogotobogo.com site search:

Introduction

In this post, we'll learn how to deploy Apache Cassandra on Kubernetes by using Minikube.

Apache Cassandra is a massively scalable open source NoSQL database.

Leveraging Kubernetes concepts such as PersistentVolume and StatefulSets, we can provide a resilient installation of Cassandra and be confident that its data (state) are safe.

In our example, a custom Cassandra seed provider lets the database discover new Cassandra instances as they join the Cassandra cluster.

We'll also see that StatefulSets make it easier to deploy stateful applications into our Kubernetes cluster.

Picture source: Classic Cassandra and Coordination

In this post, the Pods that belong to the StatefulSet are Cassandra nodes and are members of the Cassandra cluster (called a ring). When those Pods run in our Kubernetes cluster, the Kubernetes control plane schedules those Pods onto Kubernetes Nodes.

When a Cassandra node starts, it uses a seed list to bootstrap discovery of other nodes in the ring. We'll deploy a custom Cassandra seed provider that lets the database discover new Cassandra Pods as they appear inside our Kubernetes cluster.

Let's begin with starting minikube.

Minikube defaults to 1024MiB of memory and 1 CPU. Running Minikube with the default resource configuration may result in insufficient resource errors.

$ minikube start --memory 5120 --cpus=4 
Stopping "minikube" in hyperkit ...
Node "m01" stopped.
Kihyucks-MacBook-Air:Cassandra kihyuckhong$ 
Kihyucks-MacBook-Air:Cassandra kihyuckhong$ 
Kihyucks-MacBook-Air:Cassandra kihyuckhong$ minikube start --memory 5120 --cpus=4 
minikube v1.9.2 on Darwin 10.13.3
KUBECONFIG=/Users/kihyuckhong/.kube/config
Using the hyperkit driver based on existing profile
Starting control plane node m01 in cluster minikube
Restarting existing hyperkit VM for "minikube" ...
Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
Enabling addons: dashboard, default-storageclass, storage-provisioner
Done! kubectl is now configured to use "minikube"

Creating a headless Service for Cassandra

We will utilize a headless service for Cassandra so that we can provide a way for applications to access it via KubeDNS and not expose it to the outside world. To access it from our developer workstation we can use kubectl exec commands against any of the cassandra pods. If we do wish to connect an application to it we can use the KubeDNS value of cassandra.default.svc.cluster.local when configuring our application.

Picture source: Scalable multi-node Cassandra deployment on Kubernetes Cluster

The following Service (cassandra-service.yaml) is used for DNS lookups between Cassandra Pods and clients within our cluster:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra

To allow us to do simple discovery of the cassandra seed node (which we will deploy shortly) we can create a headless service. We do this by specifying None for the clusterIP in the cassandra-service.yaml. This headless service allows us to use KubeDNS for the Pods to discover the IP address of the Cassandra seed.

Let's create a Service to track all Cassandra StatefulSet members from the cassandra-service.yaml file:

$ kubectl apply -f cassandra-service.yaml
service/cassandra created

$ kubectl get svc cassandra
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   None         <none>        9042/TCP   25s

Because Cassandra is a database, we may want to use Stateful sets and Persistent Volumes to ensure resiliency in our database.

Creating a Cassandra ring using a StatefulSet

The following StatefulSet manifest (cassandra-statefuleset.yaml) that uses the default provisioner for Minikube creates a Cassandra ring that consists of three Pods:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 1
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-ssd

The StatefulSet is responsible for creating the Pods. It provides ordered deployment, ordered termination and unique network names. Run the following command to start a single Cassandra server:

$ kubectl apply -f cassandra-statefulset.yaml  
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created

We can check if our StatefulSet has deployed using the command below:

$ kubectl get statefulsets cassandra    
NAME        READY   AGE
cassandra   1/1     8m48s

Get the pods. In this case, we have only one pod and we cannot see the ordered ceration with this "replicaset=1":

$ kubectl get pods -l="app=cassandra"
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          6m24s

The Pod in this post uses the gcr.io/google-samples/cassandra:v13 image from Google’s container registry. The Docker image is based on debian-base and includes OpenJDK 8. This image includes a standard Cassandra installation from the Apache Debian repo.

Run the Cassandra nodetool inside the first Pod, to display the status of the ring:

$ kubectl exec -it cassandra-0 -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.6  99.59 KiB  32           100.0%            8b22647a-81a6-4f07-98b4-1f7cd0efd13a  Rack1-K8Demo

Clean up

To delete everything in the Cassandra StatefulSet:

$ kubectl delete statefulset -l app=cassandra 
statefulset.apps "cassandra" deleted

$ kubectl delete persistentvolumeclaim -l app=cassandra
persistentvolumeclaim "cassandra-data-cassandra-0" deleted

To delete the Service you set up for Cassandra:

$ kubectl delete service -l app=cassandra   
service "cassandra" deleted

To clean up the minikube itself:

$ minikube delete
Deleting "minikube" in hyperkit ...
Removed all traces of the "minikube" cluster.

Refs

Explains why we need the stateful set, MySQL example: Kubernetes Tutorial: Why do you need StatefulSets in Kubernetes?
Explains difference between regular pod deploy (stateless) and the stateful set deploy, MySQL example: Kubernetes StatefulSet simply explained | Deployment vs StatefulSet

The two main characteristics of a statefuleset compared with a regular deployment:

"Stateful applications are not perfect for containerized environment"