Docker & Kubernetes : Nginx Ingress Controller on GCP Kubernetes

bogotobogo.com site search:

Nginx Ingress Controller

It is built around the Kubernetes Ingress resource, using a ConfigMap to store the NGINX configuration.

Ingress allows external users and outside client applications access to HTTP services. Ingress consists of two components: Ingress resource and Ingress controller and it is vital that both pieces are properly configured so that traffic can be routed from an outside client to a Kubernetes Service.

Ingress with NGINX controller on Google Kubernetes Engine

Ingress resource is a collection of rules for the inbound traffic to reach Services. These are Layer 7 (L7) rules that allow hostnames (and optionally paths) to be directed to specific Services in Kubernetes.
Ingress controller acts upon the rules set by the Ingress Resource, typically via an HTTP or L7 load balancer.

In this post, we're going to configure a Kubernetes deployment with an Ingress Resource with NGINX as an Ingress Controller:

Deploy a simple Kubernetes web application.
Deploy an NGINX Ingress Controller using a stable Helm Chart.
Deploy an Ingress Resource for the application that uses NGINX Ingress as the controller.
Test NGINX Ingress functionality by accessing the Google Cloud L4 (TCP/UDP) Load Balancer frontend IP and ensure it can access the web application.

Google Cloud Shell

Google Cloud Shell is loaded with development tools and it offers a persistent 5GB home directory and runs on the Google Cloud. Google Cloud Shell provides command-line access to our GCP resources. We can activate the shell: in GCP console, on the top right toolbar, click the Open Cloud Shell button:

In the dialog box that opens, click "START CLOUD SHELL".

gcloud is the command-line tool for Google Cloud Platform. It comes pre-installed on Cloud Shell and supports tab-completion.

Set our zone:

$ gcloud config set compute/zone us-central1-a
Updated property [compute/zone].

Run the following command to create a Kubernetes cluster:

$ gcloud container clusters create nginx-demo --num-nodes 2
...
kubeconfig entry generated for nginx-demo.
NAME        LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
nginx-demo  us-central1-a  1.11.6-gke.2    35.224.213.155  n1-standard-1  1.11.6-gke.2  2          RUNNING

Install Helm

Helm is a tool that streamlines Kubernetes application installation and management. We can think of it as apt, yum, or homebrew for Kubernetes. It consists of two parts: Tiller(server-runs inside Kubernetes cluster and manages releases (installations) of our Helm Charts) and Helm (client).

Helm comes preconfigured with an installer script that automatically grabs the latest version of the Helm client and installs it locally. So, let's get the script:

$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  7234  100  7234    0     0  37172      0 --:--:-- --:--:-- --:--:-- 37288

Install the Helm client:

$ chmod 700 get_helm.sh

$ ./get_helm.sh
Downloading https://kubernetes-helm.storage.googleapis.com/helm-v2.12.3-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/local/bin
helm installed into /usr/local/bin/helm
tiller installed into /usr/local/bin/tiller
Run 'helm init' to configure helm.

$ helm init
...
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
...
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
...

Install Tiller

RBAC is enabled by default with Kubernetes v1.8+. Prior to installing tiller, we need to ensure that we have the correct "ServiceAccount" and "ClusterRoleBinding" configured for the tiller service. This allows tiller to be able to install services in the default namespace.

Let's install the server-side tiller with RBAC enabled:

$ kubectl create serviceaccount --namespace kube-system tiller
serviceaccount "tiller" created

$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io "tiller-cluster-rule" created


$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
deployment.extensions "tiller-deploy" patched

Now, let's initialize Helm with our newly-created service account:

$ helm init --service-account tiller --upgrade
...
Tiller (the Helm server-side component) has been upgraded to the current version.
...

We can see if tiller is running by checking for the tiller_deploy Deployment in the kube-system namespace:

$ kubectl get deployments -n kube-system
NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
event-exporter-v0.2.3    1         1         1            1           5m
fluentd-gcp-scaler       1         1         1            1           5m
heapster-v1.6.0-beta.1   1         1         1            1           5m
kube-dns                 2         2         2            2           5m
kube-dns-autoscaler      1         1         1            1           5m
l7-default-backend       1         1         1            1           5m
metrics-server-v0.2.1    1         1         1            1           5m
tiller-deploy            1         1         1            1           1m

Deploying an app

Now that we have Helm configured, let's deploy a simple web-based application from the Google Cloud Repository. This application will be used as the backend for the Ingress:

$ kubectl run hello-app --image=gcr.io/google-samples/hello-app:1.0 --port=8080
deployment.apps "hello-app" created

Now expose the hello-app Deployment as a Service:

$ kubectl expose deployment hello-app
service "hello-app" exposed

Deploying an NGINX Ingress Controller via Helm

The NGINX controller must be exposed for external access. This is done using Service type: LoadBalancer on the NGINX controller service.

On Kubernetes Engine, this creates a Google Cloud Network (TCP/IP) Load Balancer with NGINX controller Service as a backend.

Google Cloud also creates the appropriate firewall rules within the Service's VPC to allow web HTTP(S) traffic to the load balancer frontend IP address.

Let's deploy the NGINX Ingress Controller:

$ helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true
NAME:   nginx-ingress
LAST DEPLOYED: Mon Feb 18 16:26:00 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/ClusterRoleBinding
NAME           AGE
nginx-ingress  0s
==> v1beta1/RoleBinding
NAME           AGE
nginx-ingress  0s
==> v1beta1/Deployment
NAME                           DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
nginx-ingress-controller       1        1        1           0          0s
nginx-ingress-default-backend  1        1        1           0          0s
==> v1beta1/ClusterRole
NAME           AGE
nginx-ingress  0s
==> v1/ServiceAccount
NAME           SECRETS  AGE
nginx-ingress  1        0s
==> v1beta1/Role
NAME           AGE
nginx-ingress  0s
==> v1/Service
NAME                           TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
nginx-ingress-controller       LoadBalancer  10.63.243.6    <pending>    80:32654/TCP,443:31594/TCP  1s
nginx-ingress-default-backend  ClusterIP     10.63.252.116  <none>       80/TCP                      1s
==> v1beta1/Deployment
NAME                           DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
nginx-ingress-controller       1        1        1           0          1s
nginx-ingress-default-backend  1        1        1           0          1s
==> v1/Pod(related)
NAME                                            READY  STATUS             RESTARTS  AGE
nginx-ingress-controller-6b464c8b8-94b8g        0/1    ContainerCreating  0         0s
nginx-ingress-default-backend-544cfb69fc-skslh  0/1    ContainerCreating  0         0s
==> v1/ServiceAccount
NAME           SECRETS  AGE
nginx-ingress  1        1s
==> v1beta1/Role
NAME           AGE
nginx-ingress  1s

NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-ingress-controller'
An example Ingress that makes use of the controller:
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

Note the second service, nginx-ingress-default-backend. The default backend is a Service which handles all URL paths and hosts the NGINX controller. The default backend exposes two URLs:

/healthz that returns 200
/ that returns 404

We may want to wait a few moments while the GCP L4 Load Balancer gets deployed. Confirm that the nginx-ingress-controller Service has been deployed and that we have an external IP address associated with the service by running the following command:

$ kubectl get service nginx-ingress-controller
NAME                       TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-controller   LoadBalancer   10.63.243.6   35.194.29.243   80:32654/TCP,443:31594/TCP   5m

Configure Ingress Resource

An Ingress Resource object is a collection of L7 rules for routing inbound traffic to Kubernetes Services. Multiple rules can be defined in one Ingress Resource or they can be split up into multiple Ingress Resource manifests.

The Ingress Resource also determines which controller to utilize to serve traffic.

This can be set with an annotation, kubernetes.io/ingress.class, in the metadata section of the Ingress Resource.

For the NGINX controller, we will use the nginx value as shown below:

$ annotations: kubernetes.io/ingress.class: nginx

On Kubernetes Engine, if no annotation is defined under the metadata section, the Ingress Resource uses the GCP GCLB L7 load balancer to serve traffic.

Let's create a simple Ingress Resource YAML file which uses the NGINX Ingress Controller and has one path rule defined in ingress-resource.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /hello
        backend:
          serviceName: hello-app
          servicePort: 8080

In the file, The kind: Ingress dictates it is an Ingress Resource object. This Ingress Resource defines an inbound L7 rule for path /hello to service hello-app on port 8080.

To apply those rules to our Kubernetes application:

$ kubectl apply -f ingress-resource.yaml
ingress.extensions "ingress-resource" created

Verify that Ingress Resource has been created:

$ kubectl get ingress ingress-resource
NAME               HOSTS     ADDRESS        PORTS     AGE
ingress-resource   *         35.202.54.69   80        55s

Testing Ingress

Now we should be able to access the web application by going to the EXTERNAL-IP/hello address of the NGINX ingress controller:

$ kubectl get service nginx-ingress-controller
NAME                       TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-controller   LoadBalancer   10.63.243.6   35.194.29.243   80:32654/TCP,443:31594/TCP   11m

To check if the default-backend service is working properly, access any path (other than the path /hello defined in the Ingress Resource) and ensure that we receive a 404 message: