Docker: Load Testing with Locust on GCP Kubernetes

bogotobogo.com site search:

Introduction

In this post, we'll use Locust which is one of the most solidly proven performance frameworks (JMeter is another one that's likely popular). Locust is highly scalable due to its fully event-based implementation. Because of these facts, Locust has a fast-growing community, who prefer this framework over JMeter.

Locust is a distributed, Python-based load testing tool that is capable of distributing requests across multiple target paths. For example, Locust can distribute requests to the /login and /metrics target paths.

Setup GCE and Enable Cloud Shell

Setup the zone in Cloud Shell:

$ gcloud config set compute/zone us-central1-a
Updated property [compute/zone].

Get the source code from the repository:

$ git clone https://github.com/GoogleCloudPlatform/distributed-load-testing-using-kubernetes.git

$ cd distributed-load-testing-using-kubernetes/

Deploy Web Application

The sample-webapp folder contains a simple Google App Engine Python application as the "system under test". To deploy the application, we can use the gcloud app deploy command:

$ gcloud app deploy sample-webapp/app.yaml
 [1] asia-east2    (supports standard and flexible)
 [2] asia-northeast1 (supports standard and flexible)
 [3] asia-south1   (supports standard and flexible)
 [4] australia-southeast1 (supports standard and flexible)
 [5] europe-west   (supports standard and flexible)
 [6] europe-west2  (supports standard and flexible)
 [7] europe-west3  (supports standard and flexible)
 [8] northamerica-northeast1 (supports standard and flexible)
 [9] southamerica-east1 (supports standard and flexible)
 [10] us-central    (supports standard and flexible)
 [11] us-east1      (supports standard and flexible)
 [12] us-east4      (supports standard and flexible)
 [13] us-west2      (supports standard and flexible)
 [14] cancel
Please enter your numeric choice:  Please enter a value between 1 and 14:  10
...

You can stream logs from the command line by running:
  $ gcloud app logs tail -s default
To view your application in the web browser run:
  $ gcloud app browse

Create Kubernetes Engine cluster

Let's create our cluster:

$ gcloud container clusters create locust-testing
kubeconfig entry generated for locust-testing.
NAME            LOCATION       MASTER_VERSION  MASTER_IP      MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
locust-testing  us-central1-a  1.11.6-gke.2    35.225.199.73  n1-standard-1  1.11.6-gke.2  3          RUNNING

$ cd kubernetes-config/

$ ls
locust-master-controller.yaml  locust-master-service.yaml  locust-worker-controller.yaml

Deploy Locust master

The first component of the deployment is the Locust master, which is the entry point for executing the load testing tasks. The Locust master is deployed as a replication controller with a single replica because we need only one master. A replication controller is useful even when deploying a single pod because it ensures high availability.

The configuration for the replication controller specifies several elements, including the name of the controller (locust-master), labels for organization (name: locust, role: master), and the ports that need to be exposed by the container (8089 for web interface, 5557 and 5558 for communicating with workers). This information is later used to configure the Locust workers controller.

The following file (locust-master-controller.yaml) contains the configuration for the ports:

kind: ReplicationController
apiVersion: v1
metadata:
  name: locust-master
  labels:
    name: locust
    role: master
spec:
  replicas: 1
  selector:
    name: locust
    role: master
  template:
    metadata:
      labels:
        name: locust
        role: master
    spec:
      containers:
        - name: locust
          image: gcr.io/cloud-solutions-images/locust-tasks:latest
          env:
            - name: LOCUST_MODE
              value: master
            - name: TARGET_HOST
              value: http://workload-simulation-webapp.appspot.com
          ports:
            - name: loc-master-web
              containerPort: 8089
              protocol: TCP
            - name: loc-master-p1
              containerPort: 5557
              protocol: TCP
            - name: loc-master-p2
              containerPort: 5558
              protocol: TCP

Let's deploy the locust-master-controller:

$ kubectl create -f locust-master-controller.yaml
replicationcontroller "locust-master" created

$ kubectl get rc
NAME            DESIRED   CURRENT   READY     AGE
locust-master   1         1         1         1m

$ kubectl get pods -l name=locust,role=master
NAME                  READY     STATUS    RESTARTS   AGE
locust-master-kg482   1/1       Running   0          1m

Let's deploy the locust-master-service:

$ kubectl create -f locust-master-service.yaml
service "locust-master" created

This will expose the Pod with an internal DNS name (locust-master) and ports 8089, 5557, and 5558. As part of this step, the type: LoadBalancer directive in locust-master-service.yaml will tell Google Kubernetes Engine to create a Google Compute Engine forwarding-rule from a publicly avaialble IP address to the locust-master Pod.

To view the newly created forwarding-rule:

$ kubectl get svc locust-master
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                        AGE
locust-master   LoadBalancer   10.35.253.55   35.202.16.162   8089:31521/TCP,5557:31809/TCP,5558:30194/TCP   58s

Deploy locust-worker

The Locust workers execute the load testing tasks. The Locust workers are deployed by a single replication controller that creates ten pods. The pods are spread out across the Kubernetes cluster. Each pod uses environment variables to control important configuration information such as the hostname of the system under test and the hostname of the Locust master.

The following file ( contains the configuration for the name, labels, and number of replicas:

kind: ReplicationController
apiVersion: v1
metadata:
  name: locust-worker
  labels:
    name: locust
    role: worker
spec:
  replicas: 10
  selector:
    name: locust
    role: worker
  template:
    metadata:
      labels:
        name: locust
        role: worker
    spec:
      containers:
        - name: locust
          image: gcr.io/cloud-solutions-images/locust-tasks:latest
          env:
            - name: LOCUST_MODE
              value: worker
            - name: LOCUST_MASTER
              value: locust-master
            - name: TARGET_HOST
              value: http://workload-simulation-webapp.appspot.com

Let's deploy locust-worker-controller:

$ kubectl create -f locust-worker-controller.yaml
replicationcontroller "locust-worker" created

The locust-worker-controller is set to deploy 10 locust-worker Pods. Let's check if they were deployed:

$ kubectl get pods -l name=locust,role=worker
NAME                  READY     STATUS    RESTARTS   AGE
locust-worker-4zwvz   1/1       Running   0          36s
locust-worker-7bgnb   1/1       Running   0          36s
locust-worker-7jpl8   1/1       Running   0          36s
locust-worker-7qtkl   1/1       Running   0          36s
locust-worker-bfv5r   1/1       Running   0          36s
locust-worker-qt4bm   1/1       Running   0          36s
locust-worker-w69ng   1/1       Running   0          36s
locust-worker-wdh99   1/1       Running   0          36s
locust-worker-xpwqh   1/1       Running   0          36s
locust-worker-zcz6m   1/1       Running   0          36s

To scale the number of locust-worker Pods, issue a replication controller scale command:

$ kubectl scale --replicas=20 replicationcontrollers locust-worker
replicationcontroller "locust-worker" scaled

To confirm that the Pods have launched and are ready, get the list of locust-worker Pods:

$ kubectl get pods -l name=locust,role=worker
NAME                  READY     STATUS    RESTARTS   AGE
NAME                  READY     STATUS    RESTARTS   AGE
locust-worker-2x8kv   0/1       Pending   0          3m
locust-worker-4zwvz   1/1       Running   0          4m
locust-worker-5bw4z   0/1       Pending   0          3m
locust-worker-6qgqc   0/1       Pending   0          3m
locust-worker-7bgnb   1/1       Running   0          4m
locust-worker-7jpl8   1/1       Running   0          4m
locust-worker-7qtkl   1/1       Running   0          4m
locust-worker-b6t6h   0/1       Pending   0          3m
locust-worker-bfv5r   1/1       Running   0          4m
locust-worker-clh56   0/1       Pending   0          3m
locust-worker-drt55   0/1       Pending   0          3m
locust-worker-nrs6t   1/1       Running   0          3m
locust-worker-qt4bm   1/1       Running   0          4m
locust-worker-vcxm2   0/1       Pending   0          3m
locust-worker-w69ng   1/1       Running   0          4m
locust-worker-wdh99   1/1       Running   0          4m
locust-worker-wh5sd   0/1       Pending   0          3m
locust-worker-wnpvk   1/1       Running   0          3m
locust-worker-xpwqh   1/1       Running   0          4m
locust-worker-zcz6m   1/1       Running   0          4m

Execute Tests

To execute the Locust tests, get the external IP address by following command:

$ kubectl get svc locust-master
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                        AGE
locust-master   LoadBalancer   10.35.253.55   35.202.16.162   8089:31521/TCP,5557:31809/TCP,5558:30194/TCP   11m

Specify the total number of users to simulate and a rate at which each user should be spawned. Next, click Start swarming to begin the simulation.

For example we can specify number of users as 300 and rate as 10.

Click Start swarming:

As time progress and users are spawned, we will see statistics begin to aggregate for simulation metrics, such as the number of requests and requests per second: