Docker & Kubernetes : Terraform and AWS EKS

bogotobogo.com site search:

AWS EKS

The Amazon EKS service allows us to manage Kubernetes servers. We will see how to create/destroy a sample Kubernetes architecture in AWS using Terraform.

The sample architecture includes the following resources:

EKS Cluster: AWS managed Kubernetes cluster of master servers
AutoScaling Group containing 2 m4.large instances based on the latest EKS Amazon Linux 2 AMI: Operator managed Kubernetes worker nodes for running Kubernetes service deployments
Associated VPC, Internet Gateway, Security Groups, and Subnets: Operator managed networking resources for the EKS Cluster and worker node instances
Associated IAM Roles and Policies: Operator managed access resources for EKS and worker node instances

Terraform Module

We can create Kubernetes cluster using Getting Started with AWS EKS, however, in this post, we'll use community created Terraform module to launch and configure our EKS cluster and nodes.

Let's clone the community terraform module for EKS, terraform-aws-eks:

$ git clone https://github.com/WesleyCharlesBlake/terraform-aws-eks.git

$ cd terraform-aws-eks

Prerequisites

Let's make sure we have a working Terraform binary:

$ terraform version
Terraform v0.11.10
+ provider.aws v1.48.0
+ provider.http v1.0.1

Your version of Terraform is out of date! The latest version
is 0.11.10. You can update by downloading from www.terraform.io/downloads.html

If not installed, we need to download the Terraform binary (Download Terraform). On Mac:

$ brew update && brew install terraform

The Heptio Authenticator is used to integrate our AWS IAM settings with our Kubernetes RBAC permissions.

Picture from https://aws.amazon.com/blogs/opensource/deploying-heptio-authenticator-kops/

To install this, run the following from a terminal:

$ curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/darwin/amd64/aws-iam-authenticator

Apply execute permissions to the binary:

$ chmod +x ./aws-iam-authenticator

Copy the binary to a folder in the $PATH. We recommend creating a $HOME/bin/aws-iam-authenticator and ensuring that $HOME/bin comes first in the $PATH:

$ cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH

Add $HOME/bin to your PATH environment variable. For Bash shells on macOS:

$ echo 'export PATH=$HOME/bin:$PATH' >> ~/.bash_profile

Test that the aws-iam-authenticator binary works:

$ source ~/.bash_profile

$ aws-iam-authenticator help
A tool to authenticate to Kubernetes using AWS IAM credentials
...

AWS provider

In order to create an EKS cluster using Terraform, we need to configure the AWS provider by adding the provider to the Terraform configuration file (provider.tf):

provider "aws" {
  region     = "${var.aws_region}"
  profile    = "eks"
}

The region variable is stored in vars.tf:

variable "aws_region" {
  description = "US EAST Virginia"
  default     = "us-east-1"
}

Actually, the provider needs AWS credentials via environment variables:

export AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID_HERE}
export AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY_HERE}

If the env vars are not provided, Terraform will get them from credentials file, $HOME/.aws/credentials:

[eks]
aws_access_key_id = AK...5Q
aws_secret_access_key = KL...jb

Launch EKS Cluster

Let's start by initializing the Terraform state:

$ terraform init
Initializing modules...
- module.eks
...
* provider.aws: version = "~> 1.48"
* provider.http: version = "~> 1.0"

Terraform has been successfully initialized!
...

Plan our deployment:

$ terraform plan -var 'cluster-name=eks-demo-cluster' -var 'desired-capacity=2' -out eks-demo-cluster-output
...
Terraform will perform the following actions:

  + module.eks.aws_autoscaling_group.eks
  + module.eks.aws_eks_cluster.eks
  + module.eks.aws_iam_instance_profile.node
  + module.eks.aws_iam_role.cluster
  + module.eks.aws_iam_role.node
  + module.eks.aws_iam_role_policy_attachment.cluster-AmazonEKSClusterPolicy
  + module.eks.aws_iam_role_policy_attachment.cluster-AmazonEKSServicePolicy
  + module.eks.aws_iam_role_policy_attachment.node-AmazonEC2ContainerRegistryReadOnly
  + module.eks.aws_iam_role_policy_attachment.node-AmazonEKSWorkerNodePolicy
  + module.eks.aws_iam_role_policy_attachment.node-AmazonEKS_CNI_Policy
  + module.eks.aws_internet_gateway.eks
  + module.eks.aws_launch_configuration.eks
  + module.eks.aws_route_table.eks
  + module.eks.aws_route_table_association.eks[0]
  + module.eks.aws_route_table_association.eks[1]
  + module.eks.aws_security_group.cluster
  + module.eks.aws_security_group.node
  + module.eks.aws_security_group_rule.cluster-ingress-node-https
  + module.eks.aws_security_group_rule.cluster-ingress-workstation-https
  + module.eks.aws_security_group_rule.node-ingress-cluster
  + module.eks.aws_security_group_rule.node-ingress-self
  + module.eks.aws_subnet.eks[0]
  + module.eks.aws_subnet.eks[1]
  + module.eks.aws_vpc.eks
 ...
Plan: 24 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: eks-demo-cluster-output

To perform exactly these actions, run the following command to apply:
    terraform apply "eks-demo-cluster-output"

Apply that plan:

$ terraform apply "eks-demo-cluster-output"

Create Kubeconfig File

Once the cluster running, we need to create the KubeConfig file that will be used to manage the cluster.

The terraform module stores the kubeconfig information in it's state store. We can view it with the following command:

$ terraform output kubeconfig

Actually, we saw the same output from the "terraform apply ..." command.

And we can save it for use with the following command:

$ terraform output kubeconfig > ${HOME}/.kube/config-eks-demo-cluster-tf

Then, we may want to add this new config to the kubectl config list:

$ export KUBECONFIG=${HOME}/.kube/config-eks-demo-cluster-tf:${HOME}/.kube/config

$ echo "export KUBECONFIG=${KUBECONFIG}" >> ${HOME}/.bash_profile

Update the Kube config:

$ aws eks update-kubeconfig --name eks-demo-cluster
Added new context arn:aws:eks:us-east-1:***:cluster/eks-demo-cluster to /Users/kihyuckhong/.kube/config-eks-demo-cluster-tf

Create the Worker ConfigMap

The terraform state also contains a config-map we can use for our EKS workers.

View the configmap:

$ terraform output config-map
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::***:role/eks-demo-cluster-eks-node-role
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

Save the config-map:

$ terraform output config-map > /tmp/config-map-aws-auth.yml

Apply the config-map:

$ kubectl apply -f /tmp/config-map-aws-auth.yml
configmap/aws-auth created

Confirm our cluster Nodes:

$ kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
ip-10-0-0-173.ec2.internal   Ready    <none>   2m    v1.10.3
ip-10-0-1-97.ec2.internal    Ready    <none>   2m    v1.10.3

We now have a fully working Amazon EKS Cluster that is ready to use!

Kubernetes Dashboard deploy

The official Kubernetes dashboard is not deployed by default. However, we can deploy the dashboard with the kubectl command:

$ kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created

Since this is deployed to our private cluster, we need to access it via a proxy. Kube-proxy is available to proxy our requests to the dashboard service. Let's run the following command:

$ kubectl proxy --port=8080 --address='0.0.0.0' --disable-filter=true
W1126 16:14:15.144712   63279 proxy.go:140] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious
Starting to serve on [::]:8080

This will start the proxy, listen on port 8080, listen on all interfaces, and will disable the filtering of non-localhost requests.

While leaving the proxy running in our current terminal tab, we may want to open a new terminal tab to continue.

Granting Cluster-Wide Admin Permission to the Dashboard

Authentication is normally handled by reading the local kubectl file, but our kubectl authentication is plugin based. So, we are going to grant Admin privileges to the Dashboard Service Account while granting admin privileges to Dashboard's Service Account might be a security risk.

First we will deploy the deployment file, dashboard-admin.yaml:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

We can now deploy it to our cluster:

$ kubectl create -f dashboard-admin.yaml
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created

The dashboard can now be accessed from:

http://localhost:8080/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

Select "Skip":

RDS Postgres

We need RDS Postgres since we're not containerizing it in this post.

See how we want to set it up : Docker & Kubernetes 4 : Django with RDS via AWS Kops

Deploying Django app

Our Django app can be deployed with "kubectl" command:

$ cd deploy

$ kubectl apply -f rds/
secret/postgres-credentials created
service/postgres-service created

$ kubectl apply -f redis/
deployment.apps/redis created
service/redis-service created

$ kubectl apply -f django/
deployment.apps/django created
job.batch/django-migrations created
service/django-service created

$ kubectl apply -f celery/
deployment.apps/celery-beat created
deployment.apps/celery-worker created

Now we can see the container and pods from Kube dashboard.

How can we access our Django App?

We need to retrieve the Load balancer endpoint which is in the row of the django-service. Just click it.

Cleanup

To delete the resources created for this EKS cluster, run the following commands.

View the plan:

$ terraform plan -destroy -out eks-demo-cluster-destroy-tf

Execute the plan:

$ terraform apply "eks-demo-cluster-destroy-tf"

References

terraform modules:

Amazon EKS Workshop > Launch using Terraform > Get Terraform Module

I used the module but got an error of getting ipv6 instead of ipv4 address:

* module.eks.aws_security_group_rule.cluster-ingress-workstation-https: "cidr_blocks.0" must contain a valid network CIDR, got "2600:1700:38d0:3e60:8c72:8bc4:ab29:80bc/32"

Fix (modules/eks/workstation-external-ip.tf):

# Workstation External IP

data "http" "workstation-external-ip" {
  #url = "http://icanhazip.com" . =>
  url = "http://ipv4.icanhazip.com"

}

locals {
  workstation-external-cidr = "${chomp(data.http.workstation-external-ip.body)}/32"
}

The fixed module and Django app are available from terraform-aws-eks/

Note: got another error which appeared to be related to the instance type and cni. So, I switched the type to t2.medium from t2.micro:

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "django-migrations-s6m64_default" network: add cmd: failed to assign an IP address to container