Docker & Kubernetes: Horizontal pod autoscaler on minikube

bogotobogo.com site search:

Prerequisites

metrics-server monitoring needs to be deployed in the cluster to provide metrics via the resource metrics API, as Horizontal Pod Autoscaler uses this API to collect metrics:

$ minikube addons enable metrics-server
The 'metrics-server' addon is enabled

Check it's been enabled:

$ minikube addons list
|-----------------------------|----------|--------------|
|         ADDON NAME          | PROFILE  |    STATUS    |
|-----------------------------|----------|--------------|
| ambassador                  | minikube | disabled     |
| dashboard                   | minikube | enabled ✅   |
| default-storageclass        | minikube | enabled ✅   |
| efk                         | minikube | disabled     |
| freshpod                    | minikube | disabled     |
| gvisor                      | minikube | disabled     |
| helm-tiller                 | minikube | disabled     |
| ingress                     | minikube | disabled     |
| ingress-dns                 | minikube | disabled     |
| istio                       | minikube | disabled     |
| istio-provisioner           | minikube | disabled     |
| kubevirt                    | minikube | disabled     |
| logviewer                   | minikube | disabled     |
| metallb                     | minikube | disabled     |
| metrics-server              | minikube | enabled ✅   |
| nvidia-driver-installer     | minikube | disabled     |
| nvidia-gpu-device-plugin    | minikube | disabled     |
| olm                         | minikube | disabled     |
| pod-security-policy         | minikube | disabled     |
| registry                    | minikube | disabled     |
| registry-aliases            | minikube | disabled     |
| registry-creds              | minikube | disabled     |
| storage-provisioner         | minikube | enabled ✅   |
| storage-provisioner-gluster | minikube | disabled     |
|-----------------------------|----------|--------------|

Deploy Apache server

The index page performs some CPU intensive computations:

<?php
  $x = 0.0001;
  for ($i = 0; $i <= 1000000; $i++) {
    $x += sqrt($x);
  }
  echo "OK!";
?>

Deploy:

$ kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
deployment.apps/php-apache created
service/php-apache created

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
php-apache-5c4f475bf5-nql2n   1/1     Running   0          56s

Create Horizontal Pod Autoscaler

Using kubectl autoscale, we will create the autoscaler.

The following command will create a Horizontal Pod Autoscaler (HPA) that maintains between 1 and 10 replicas of the Pods controlled by the php-apache deployment we created in the previous step.

Via the deployment, HPA will increase and decrease the number of replicas to maintain an average CPU utilization across all Pods of 50% (since each pod requests 200 milli-cores by kubectl run), this means average CPU usage of 100 milli-cores).

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10  
horizontalpodautoscaler.autoscaling/php-apache autoscaled

Check the current status of autoscaler:

$ kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          2m1s

As we can see from the output, the current CPU consumption is 0%. That's because we are not sending any requests to the server.

The TARGET column shows the average across all the pods controlled by the corresponding deployment.

Increase the load and the stop the load

Now, we will see how the autoscaler reacts to increased load. We will start a container called "load generator", and send an infinite loop of queries to the php-apache service.

Run the following command in a different terminal:

$ kubectl run -it --rm load-generator --image=busybox /bin/sh
If you don't see a command prompt, try pressing enter.
/ # while true; do wget -q -O- http://php-apache; done    
OK!OK!OK!OK!OK!....

Back to the other terminal:

$ kubectl get hpa -w
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          26m
php-apache   Deployment/php-apache   234%/50%   1         10        1          27m
php-apache   Deployment/php-apache   234%/50%   1         10        4          28m
php-apache   Deployment/php-apache   234%/50%   1         10        5          28m
php-apache   Deployment/php-apache   95%/50%    1         10        5          28m
php-apache   Deployment/php-apache   56%/50%    1         10        5          29m
php-apache   Deployment/php-apache   56%/50%    1         10        6          30m
php-apache   Deployment/php-apache   50%/50%    1         10        6          30m
php-apache   Deployment/php-apache   50%/50%   1         10        6          31m
php-apache   Deployment/php-apache   48%/50%   1         10        6          31m

Here, CPU consumption has increased to 234% of the request (but it must have been over 250%). As a result, the deployment was resized to 6 replicas.

Let's stop the load.

In the terminal where we created the container with busybox image, terminate the load generation by typing <Ctrl> + C. Then we will verify the result state (after a while):

$ kubectl get hpa -w
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
...
php-apache   Deployment/php-apache   47%/50%   1         10        6          32m

php-apache   Deployment/php-apache   0%/50%    1         10        6          33m


php-apache   Deployment/php-apache   0%/50%    1         10        6          34m



php-apache   Deployment/php-apache   0%/50%    1         10        6          38m
php-apache   Deployment/php-apache   -82%/50%   1         10        1          38m
php-apache   Deployment/php-apache   0%/50%     1         10        1          39m    


$ kubectl get deployment php-apache
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
php-apache   1/1     1            1           18m

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
load-generator                1/1     Running   0          8m32s
php-apache-5c4f475bf5-x75f8   1/1     Running   0          22m

As we can see, the CPU utilization dropped to 0, and so HPA autoscaled the number of replicas back down to 1.

We can also specify resource metrics in terms of direct values, instead of as percentages of the requested value.

There are two other types of metrics, both of which are considered custom metrics: pod metrics and object metrics. These metrics may have names which are cluster specific, and require a more advanced cluster monitoring setup. Please check the Refs in the next section below.

Refs