Docker & Kubernetes : Deploy Prometheus and Grafana using Helm and Prometheus Operator - Monitoring Kubernetes node resources out of the box

Note: Operator for Prometheus has been deprecated. To deploy, please check out Docker & Kubernetes : Deploy Prometheus and Grafana using kube-prometheus-stack Helm Chart.

Since its intruction in 2016 by CoreOS, the operators are rapidly becoming the standard way of managing applications on Kubernetes especially the ones which are stateful in nature.

Whenever we deploy our application on Kubernetes cluster, we leverage multiple Kubernetes objects such as deployment, service, role, config map, ingress, etc. However, as our application gets complex, managing our application only with the native Kubernetes objects becomes difficult.

While Kubernetes manages a complete lifecycle of stateless application, it cannot manage (automate) the lifecycle of stateless application.

Combined with CRDs (custom resource definitions) and custom controllers, operators eliminate the requirement for manual intervention while performing such tasks as upgrading, handling failure recovery, and scaling stateful applications.

In summary, by using operators, we can simplify and standardize the management of complex applications and stateful workloads within our Kubernetes cluster. This approach aligns with the principles of GitOps and declarative infrastructure, which aim to specify the desired state of our infrastructure and applications as code.

There are great and Kubernetes ready apps in public repository.

Is any Helm chart for Prometheus with Operator?

$ helm search prometheus
NAME                                 	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/prometheus                    	11.12.0      	2.20.1     	Prometheus is a monitoring system and time series database. 
stable/prometheus-adapter            	2.5.0        	v0.7.0     	A Helm chart for k8s prometheus adapter                     
stable/prometheus-blackbox-exporter  	4.3.0        	0.16.0     	Prometheus Blackbox Exporter                                
stable/prometheus-cloudwatch-exporter	0.8.3        	0.8.0      	A Helm chart for prometheus cloudwatch-exporter             
stable/prometheus-consul-exporter    	0.1.5        	0.4.0      	A Helm chart for the Prometheus Consul Exporter             
stable/prometheus-couchdb-exporter   	0.1.1        	1.0        	A Helm chart to export the metrics from couchdb in Promet...
stable/prometheus-mongodb-exporter   	2.8.0        	v0.10.0    	A Prometheus exporter for MongoDB metrics                   
stable/prometheus-mysql-exporter     	0.7.0        	v0.11.0    	A Helm chart for prometheus mysql exporter with cloudsqlp...
stable/prometheus-nats-exporter      	2.5.0        	0.6.2      	A Helm chart for prometheus-nats-exporter                   
stable/prometheus-node-exporter      	1.11.1       	1.0.1      	A Helm chart for prometheus node-exporter                   
stable/prometheus-operator           	9.3.1        	0.38.1     	Provides easy monitoring definitions for Kubernetes servi...
stable/prometheus-postgres-exporter  	1.3.0        	0.8.0      	A Helm chart for prometheus postgres-exporter               
stable/prometheus-pushgateway        	1.4.1        	1.2.0      	A Helm chart for prometheus pushgateway                     
stable/prometheus-rabbitmq-exporter  	0.5.5        	v0.29.0    	Rabbitmq metrics exporter for prometheus                    
stable/prometheus-redis-exporter     	3.5.0        	1.3.4      	Prometheus exporter for Redis metrics                       
stable/prometheus-snmp-exporter      	0.0.5        	0.14.0     	Prometheus SNMP Exporter                                    
stable/prometheus-to-sd              	0.3.0        	0.5.2      	Scrape metrics stored in prometheus format and push them ...
stable/elasticsearch-exporter        	3.7.0        	1.1.0      	Elasticsearch stats exporter for Prometheus                 
stable/helm-exporter                 	0.3.3        	0.4.0      	DEPRECATED Exports helm release stats to prometheus         
stable/karma                         	1.5.2        	v0.68      	A Helm chart for Karma - an UI for Prometheus Alertmanager  
stable/stackdriver-exporter          	1.2.3        	0.6.0      	Stackdriver exporter for Prometheus                         
stable/weave-cloud                   	0.3.7        	1.4.0      	Weave Cloud is a add-on to Kubernetes which provides Cont...
stable/kube-state-metrics            	2.8.14       	1.9.7      	Install kube-state-metrics to generate and expose cluster...
stable/kuberhealthy                  	1.2.6        	v1.0.2     	The official Helm chart for Kuberhealthy.                   
stable/mariadb                       	7.3.14       	10.3.22    	DEPRECATED Fast, reliable, scalable, and easy to use open...    

Ok, we can use the one we found.

The following command deploys prometheus-operator on the Kubernetes cluster in the default configuration. The default installation includes Prometheus Operator, Alertmanager, Grafana, and configuration for scraping Kubernetes infrastructure.

$ helm install --name my-release stable/prometheus-operator 
NAME:   my-release
LAST DEPLOYED: Fri Aug 14 21:07:56 2020
NAMESPACE: default
STATUS: DEPLOYED
...

Done!
Everything has been deployed: pod/containers/services/secrets for Prometheus and Grafana.

Now, we can not only monitor cluster and pod health but also has default alerting systm is in place.
We have Grafana dashboard and Prometheus UI as well.

We'll get to the internal details later sections of this post.

The chart repo is available from https://github.com/helm/charts/tree/master/stable/prometheus-operator.

When we look at the deployed services, the service types are all ClusterIP types. That means they are cluster internal and we cannot access them from outside:

$ kubectl get svc
NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   47m
kubernetes                                ClusterIP   10.96.0.1        <none>        443/TCP                      63m
my-release-grafana                        ClusterIP   10.110.28.33     <none>        80/TCP                       47m
my-release-kube-state-metrics             ClusterIP   10.104.105.71    <none>        8080/TCP                     47m
my-release-prometheus-node-exporter       ClusterIP   10.104.199.145   <none>        9100/TCP                     47m
my-release-prometheus-oper-alertmanager   ClusterIP   10.103.236.215   <none>        9093/TCP                     47m
my-release-prometheus-oper-operator       ClusterIP   10.105.131.82    <none>        8080/TCP,443/TCP             47m
my-release-prometheus-oper-prometheus     ClusterIP   10.99.31.103     <none>        9090/TCP                     47m
prometheus-operated                       ClusterIP   None             <none>        9090/TCP                     46m    

We'll use kubectl port-forward which is useful for testing/debugging purposes because we can access our service locally without exposing it: Kubernetes back-end services not intended for remote exposure.

Let's get the port of the Grafana from the container:

$ kubectl get pod
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-my-release-prometheus-oper-alertmanager-0   2/2     Running   0          59m
my-release-grafana-9c8fb59b6-bgktk                       2/2     Running   0          59m
my-release-kube-state-metrics-844f86b568-6p54b           1/1     Running   0          59m
my-release-prometheus-node-exporter-vwjhq                1/1     Running   0          59m
my-release-prometheus-oper-operator-6497fd9d4f-tjtht     2/2     Running   0          59m
prometheus-my-release-prometheus-oper-prometheus-0       3/3     Running   1          59m   

$ kubectl logs my-release-grafana-9c8fb59b6-bgktk -c grafana
...
t=2020-08-15T04:09:51+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin
t=2020-08-15T04:09:51+0000 lvl=info msg="Starting plugin search" logger=plugins
t=2020-08-15T04:09:51+0000 lvl=info msg="Registering plugin" logger=plugins name="Direct Input"
t=2020-08-15T04:09:51+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Prometheus uid=
t=2020-08-15T04:09:51+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000  protocol=http subUrl= socket=
t=2020-08-15T05:06:57+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=127.0.0.1 time_ms=1 size=29 referer=

From the logs we got the default Grafana user=admin, we can get the password from https://github.com/helm/charts/tree/master/stable/prometheus-operator:

As discussed earlier, the kubectl port-forward command establishes a tunnel from the target pod to our localhost. The command requires us to define the type or name of the resource as well as local and remote port numbers:

kubectl port-forward TYPE/NAME [options] LOCAL_PORT:REMOTE_PORT

Now, we are almost ready to use Grafana. Just do a port-forward. We'll use deployment to select port-forward pod:

$ kubectl get deployment
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
my-release-grafana                    1/1     1            1           52m
my-release-kube-state-metrics         1/1     1            1           52m
my-release-prometheus-oper-operator   1/1     1            1           52m   

$ kubectl port-forward deployment/my-release-grafana  3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Login and select Manage:

Lots of monitoring dashboards are already there: API server, cluster, nodes, pods, statefulsets, etc.

$ kubectl get pod
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-my-release-prometheus-oper-alertmanager-0   2/2     Running   0          121m
my-release-grafana-9c8fb59b6-bgktk                       2/2     Running   0          122m
my-release-kube-state-metrics-844f86b568-6p54b           1/1     Running   0          122m
my-release-prometheus-node-exporter-vwjhq                1/1     Running   0          122m
my-release-prometheus-oper-operator-6497fd9d4f-tjtht     2/2     Running   0          122m
prometheus-my-release-prometheus-oper-prometheus-0       3/3     Running   1          121m

Port-forward:

$ kubectl port-forward prometheus-my-release-prometheus-oper-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090    

With PromQL:

Recall, we did helm install to deploy our Prometheus stack:

$ helm install --name my-release stable/prometheus-operator 

We don't need to do it but let's fetch the release history:

$ helm history my-release
REVISION	UPDATED                 	STATUS  	CHART                    	APP VERSION	DESCRIPTION     
1       	Sat Aug 14 22:07:39 2020	DEPLOYED	prometheus-operator-9.3.1	0.38.1     	Install complete    

To undo the deploy, we may want to use helm delete with the release name:

$ helm delete --name my-release
release "my-release" deleted

Although the release has been deleted and the Prometheus no longer runs, Helm will save all the revision information for a future re-deploy. If we trying to re-install the my-release right now, we would have an error:

$ helm install --name my-release stable/prometheus-operator 
Error: a release named my-release already exists.
Run: helm ls --all my-release; to check the status of the release
Or run: helm del --purge my-release; to delete it    

We can see the deleted releases:

$ helm ls --deleted
NAME      	REVISION	UPDATED                 	STATUS 	CHART                    	APP VERSION	NAMESPACE
my-release	1       	Fri Aug 14 21:07:56 2020	DELETED	prometheus-operator-9.3.1	0.38.1     	default 

$ helm ls --all my-release
NAME      	REVISION	UPDATED                 	STATUS 	CHART                    	APP VERSION	NAMESPACE
my-release	1       	Fri Aug 14 21:07:56 2020	DELETED	prometheus-operator-9.3.1	0.38.1     	default 

Lastly, let's purge it:

$ helm del --purge my-release
release "my-release" deleted

$ helm ls --all my-release    
$

Now, the Prometheus release (my-release) has been completely removed, and the release name can be reused.

Docker & K8s

1. Docker install on Amazon Linux AMI
2. Docker install on EC2 Ubuntu 14.04
3. Docker container vs Virtual Machine
4. Docker install on Ubuntu 14.04
5. Docker Hello World Application
6. Nginx image - share/copy files, Dockerfile
7. Working with Docker images : brief introduction
8. Docker image and container via docker commands (search, pull, run, ps, restart, attach, and rm)
9. More on docker run command (docker run -it, docker run --rm, etc.)
10. Docker Networks - Bridge Driver Network
11. Docker Persistent Storage
12. File sharing between host and container (docker run -d -p -v)
13. Linking containers and volume for datastore
14. Dockerfile - Build Docker images automatically I - FROM, MAINTAINER, and build context
15. Dockerfile - Build Docker images automatically II - revisiting FROM, MAINTAINER, build context, and caching
16. Dockerfile - Build Docker images automatically III - RUN
17. Dockerfile - Build Docker images automatically IV - CMD
18. Dockerfile - Build Docker images automatically V - WORKDIR, ENV, ADD, and ENTRYPOINT
19. Docker - Apache Tomcat
20. Docker - NodeJS
21. Docker - NodeJS with hostname
22. Docker Compose - NodeJS with MongoDB
23. Docker - Prometheus and Grafana with Docker-compose
24. Docker - StatsD/Graphite/Grafana
25. Docker - Deploying a Java EE JBoss/WildFly Application on AWS Elastic Beanstalk Using Docker Containers
26. Docker : NodeJS with GCP Kubernetes Engine
27. Docker : Jenkins Multibranch Pipeline with Jenkinsfile and Github
28. Docker : Jenkins Master and Slave
29. Docker - ELK : ElasticSearch, Logstash, and Kibana
30. Docker - ELK 7.6 : Elasticsearch on Centos 7
31. Docker - ELK 7.6 : Filebeat on Centos 7
32. Docker - ELK 7.6 : Logstash on Centos 7
33. Docker - ELK 7.6 : Kibana on Centos 7
34. Docker - ELK 7.6 : Elastic Stack with Docker Compose
35. Docker - Deploy Elastic Cloud on Kubernetes (ECK) via Elasticsearch operator on minikube
36. Docker - Deploy Elastic Stack via Helm on minikube
37. Docker Compose - A gentle introduction with WordPress
38. Docker Compose - MySQL
39. MEAN Stack app on Docker containers : micro services
40. MEAN Stack app on Docker containers : micro services via docker-compose
41. Docker Compose - Hashicorp's Vault and Consul Part A (install vault, unsealing, static secrets, and policies)
42. Docker Compose - Hashicorp's Vault and Consul Part B (EaaS, dynamic secrets, leases, and revocation)
43. Docker Compose - Hashicorp's Vault and Consul Part C (Consul)
44. Docker Compose with two containers - Flask REST API service container and an Apache server container
45. Docker compose : Nginx reverse proxy with multiple containers
46. Docker & Kubernetes : Envoy - Getting started
47. Docker & Kubernetes : Envoy - Front Proxy
48. Docker & Kubernetes : Ambassador - Envoy API Gateway on Kubernetes
49. Docker Packer
50. Docker Cheat Sheet
51. Docker Q & A #1
52. Kubernetes Q & A - Part I
53. Kubernetes Q & A - Part II
54. Docker - Run a React app in a docker
55. Docker - Run a React app in a docker II (snapshot app with nginx)
56. Docker - NodeJS and MySQL app with React in a docker
57. Docker - Step by Step NodeJS and MySQL app with React - I
58. Installing LAMP via puppet on Docker
59. Docker install via Puppet
60. Nginx Docker install via Ansible
61. Apache Hadoop CDH 5.8 Install with QuickStarts Docker
62. Docker - Deploying Flask app to ECS
63. Docker Compose - Deploying WordPress to AWS
64. Docker - WordPress Deploy to ECS with Docker-Compose (ECS-CLI EC2 type)
65. Docker - WordPress Deploy to ECS with Docker-Compose (ECS-CLI Fargate type)
66. Docker - ECS Fargate
67. Docker - AWS ECS service discovery with Flask and Redis
68. Docker & Kubernetes : minikube
69. Docker & Kubernetes 2 : minikube Django with Postgres - persistent volume
70. Docker & Kubernetes 3 : minikube Django with Redis and Celery
71. Docker & Kubernetes 4 : Django with RDS via AWS Kops
72. Docker & Kubernetes : Kops on AWS
73. Docker & Kubernetes : Ingress controller on AWS with Kops
74. Docker & Kubernetes : HashiCorp's Vault and Consul on minikube
75. Docker & Kubernetes : HashiCorp's Vault and Consul - Auto-unseal using Transit Secrets Engine
76. Docker & Kubernetes : Persistent Volumes & Persistent Volumes Claims - hostPath and annotations
77. Docker & Kubernetes : Persistent Volumes - Dynamic volume provisioning
78. Docker & Kubernetes : DaemonSet
79. Docker & Kubernetes : Secrets
80. Docker & Kubernetes : kubectl command
81. Docker & Kubernetes : Assign a Kubernetes Pod to a particular node in a Kubernetes cluster
82. Docker & Kubernetes : Configure a Pod to Use a ConfigMap
83. AWS : EKS (Elastic Container Service for Kubernetes)
84. Docker & Kubernetes : Run a React app in a minikube
85. Docker & Kubernetes : Minikube install on AWS EC2
86. Docker & Kubernetes : Cassandra with a StatefulSet
87. Docker & Kubernetes : Terraform and AWS EKS
88. Docker & Kubernetes : Pods and Service definitions
89. Docker & Kubernetes : Service IP and the Service Type
90. Docker & Kubernetes : Kubernetes DNS with Pods and Services
91. Docker & Kubernetes : Headless service and discovering pods
92. Docker & Kubernetes : Scaling and Updating application
93. Docker & Kubernetes : Horizontal pod autoscaler on minikubes
94. Docker & Kubernetes : From a monolithic app to micro services on GCP Kubernetes
95. Docker & Kubernetes : Rolling updates
96. Docker & Kubernetes : Deployments to GKE (Rolling update, Canary and Blue-green deployments)
97. Docker & Kubernetes : Slack Chat Bot with NodeJS on GCP Kubernetes
98. Docker & Kubernetes : Continuous Delivery with Jenkins Multibranch Pipeline for Dev, Canary, and Production Environments on GCP Kubernetes
99. Docker & Kubernetes : NodePort vs LoadBalancer vs Ingress
100. Docker & Kubernetes : MongoDB / MongoExpress on Minikube
101. Docker & Kubernetes : Load Testing with Locust on GCP Kubernetes
102. Docker & Kubernetes : MongoDB with StatefulSets on GCP Kubernetes Engine
103. Docker & Kubernetes : Nginx Ingress Controller on Minikube
104. Docker & Kubernetes : Setting up Ingress with NGINX Controller on Minikube (Mac)
105. Docker & Kubernetes : Nginx Ingress Controller for Dashboard service on Minikube
106. Docker & Kubernetes : Nginx Ingress Controller on GCP Kubernetes
107. Docker & Kubernetes : Kubernetes Ingress with AWS ALB Ingress Controller in EKS
108. Docker & Kubernetes : Setting up a private cluster on GCP Kubernetes
109. Docker & Kubernetes : Kubernetes Namespaces (default, kube-public, kube-system) and switching namespaces (kubens)
110. Docker & Kubernetes : StatefulSets on minikube
111. Docker & Kubernetes : RBAC
112. Docker & Kubernetes Service Account, RBAC, and IAM
113. Docker & Kubernetes - Kubernetes Service Account, RBAC, IAM with EKS ALB, Part 1
114. Docker & Kubernetes : Helm Chart
115. Docker & Kubernetes : My first Helm deploy
116. Docker & Kubernetes : Readiness and Liveness Probes
117. Docker & Kubernetes : Helm chart repository with Github pages
118. Docker & Kubernetes : Deploying WordPress and MariaDB with Ingress to Minikube using Helm Chart
119. Docker & Kubernetes : Deploying WordPress and MariaDB to AWS using Helm 2 Chart
120. Docker & Kubernetes : Deploying WordPress and MariaDB to AWS using Helm 3 Chart
121. Docker & Kubernetes : Helm Chart for Node/Express and MySQL with Ingress
122. Docker & Kubernetes : Deploy Prometheus and Grafana using Helm and Prometheus Operator - Monitoring Kubernetes node resources out of the box
123. Docker & Kubernetes : Deploy Prometheus and Grafana using kube-prometheus-stack Helm Chart
124. Docker & Kubernetes : Istio (service mesh) sidecar proxy on GCP Kubernetes
125. Docker & Kubernetes : Istio on EKS
126. Docker & Kubernetes : Istio on Minikube with AWS EC2 for Bookinfo Application
127. Docker & Kubernetes : Deploying .NET Core app to Kubernetes Engine and configuring its traffic managed by Istio (Part I)
128. Docker & Kubernetes : Deploying .NET Core app to Kubernetes Engine and configuring its traffic managed by Istio (Part II - Prometheus, Grafana, pin a service, split traffic, and inject faults)
129. Docker & Kubernetes : Helm Package Manager with MySQL on GCP Kubernetes Engine
130. Docker & Kubernetes : Deploying Memcached on Kubernetes Engine
131. Docker & Kubernetes : EKS Control Plane (API server) Metrics with Prometheus
132. Docker & Kubernetes : Spinnaker on EKS with Halyard
133. Docker & Kubernetes : Continuous Delivery Pipelines with Spinnaker and Kubernetes Engine
134. Docker & Kubernetes : Multi-node Local Kubernetes cluster : Kubeadm-dind (docker-in-docker)
135. Docker & Kubernetes : Multi-node Local Kubernetes cluster : Kubeadm-kind (k8s-in-docker)
136. Docker & Kubernetes : nodeSelector, nodeAffinity, taints/tolerations, pod affinity and anti-affinity - Assigning Pods to Nodes
137. Docker & Kubernetes : Jenkins-X on EKS
138. Docker & Kubernetes : ArgoCD App of Apps with Heml on Kubernetes
139. Docker & Kubernetes : ArgoCD on Kubernetes cluster
140. Docker & Kubernetes : GitOps with ArgoCD for Continuous Delivery to Kubernetes clusters (minikube) - guestbook