Docker and Kubernetes Headless service and discovering pods

bogotobogo.com site search:

Introduction to Headless

We know how services can be used to provide a stable IP address allowing clients to connect to pods (or endpoints). Each connection to the service is forwarded to one randomly selected backing pod. But what if the client needs to connect to all of those pods? What if the backing pods themselves need to each connect to all the other backing pods?

For a client to connect to all pods, it needs to figure out the the IP of each individual pod. Kubernetes, fortunately, allows clients to discover pod IPs through DNS lookups. Usually, when we perform a DNS lookup for a service, the DNS server returns a single IP (the service's cluster IP). But if we tell Kubernetes we don't need a cluster IP for our service by setting the clusterIP field to None in the service specification, the DNS server will return the pod IPs instead of the single service IP.

Instead of returning a single DNS A record, the DNS server will return multiple A records for the service, each pointing to the IP of an individual pod backing the service at that moment. Clients can therefore do a simple DNS A record lookup and get the IPs of all the pods that are part of the service. The client can then use that information to connect to one, many, or all of them.

Creating a headless service

Setting the clusterIP field in a service spec to None makes the service headless, and Kubernetes won't assign it a cluster IP through which clients could connect to the pods behind it.

We'll create a headless service called bogo-headless now using the following bogo-svc-headless.yaml:

apiVersion: v1
kind: Service
metadata:
  name: bogo-headless
spec:
  clusterIP: None
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: bogo

$ kubectl create -f bogo-svc-headless.yaml 
service/bogo-headless created 

$ kubectl get svc
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
bogo-headless   ClusterIP   None          <none>        80/TCP         2m6s
bogo-nodeport   NodePort    10.109.23.2   <none>        80:30123/TCP   45h
kubernetes      ClusterIP   10.96.0.1     <none>        443/TCP        45h

$ kubectl describe svc bogo-headless
$ kubectl describe svc bogo-headless
Name:              bogo-headless
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=bogo
Type:              ClusterIP
IP:                None
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.3:8080,172.17.0.5:8080,172.17.0.6:8080
Session Affinity:  None
Events:            <none>

$ kubectl get pods -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
bogo-fffth   1/1     Running   0          46h   172.17.0.3   minikube   <none>           <none>
bogo-hhxr7   1/1     Running   0          46h   172.17.0.5   minikube   <none>           <none>
bogo-qtkqj   1/1     Running   0          46h   172.17.0.6   minikube   <none>           <none>

After creating the service, we can inspect it. We see it has no cluster IP and its endpoints include the pods matching its pod selector.

Discovering pods through DNS

Now that our pods are ready, we can now try DNS lookup to see if we get the actual pod IPs or not.

We'll need to perform the lookup from inside one of the pods. However, beause our container image doesn't include the nslookup (or the dig) binary, we can't use it for the DNS lookup.

All we're trying to do is perform a DNS lookup from inside a pod running in the cluster. Why not run a new pod based on an image that contains the binaries you need? To perform DNS-related actions, you can use the tutum/dnsutils container image, which is available on Docker Hub and contains both the nslookup and the dig binaries. To run the pod, you can go through the whole process of creating a YAML manifest for it and passing it to kubectl create, but that’s too much work, right? Luckily, there’s a faster way.

$ kubectl run dnsutils --image=tutum/dnsutils --generator=run-pod/v1 \
--command -- sleep infinity    
Flag --generator has been deprecated, has no effect and will be removed in the future.
pod/dnsutils created

The --generator=run-pod/v1 option tells kubectl to create the pod directly, without any kind of ReplicationController behind it.

Let's do the DNS lookup from the newly created pod:

$ kubectl exec dnsutils -- nslookup bogo-headless
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	bogo-headless.default.svc.cluster.local
Address: 172.17.0.3
Name:	bogo-headless.default.svc.cluster.local
Address: 172.17.0.5
Name:	bogo-headless.default.svc.cluster.local
Address: 172.17.0.6

The DNS server returns three different IPs for the bogo-headless.default.svc.cluster.local FQDN.

This is different from what DNS returns for regular (non-headless) services, such as for our bogo service, where the returned IP is the service's cluster IP:

$ kubectl exec dnsutils -- nslookup bogo-nodeport
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	bogo-nodeport.default.svc.cluster.local
Address: 10.109.23.2

Although headless services may seem different from regular services, they aren't that different from the clients' perspective. Even with a headless service, clients can connect to its pods by connecting to the service's DNS name, as they can with regular services. But with headless services, because DNS returns the pods' IPs, clients connect directly to the pods, instead of through the service proxy.

A headless services still provides load balancing across pods, but through the DNS round-robin mechanism instead of through the service proxy.