AWS : Setting up Autoscaling Alarms and Notifications via CLI and Cloudformation

bogotobogo.com site search:

Introduction

We'll play with autoscaling by setting up CloudWatch alarms for low/high CPU utilization. In the process we'll create Launch configuration, autoscaling policies, etc. via CLI commands.

All of those will be performed on another instances with CLI tools with the information about autoscaling. The base instance will be launched via cloudformation which we'll see shortly in next section.

Autoscaling group will use the base AMI in its launch configuration.

Cloudformation for the base instance

Cloudformation template (ASG-env.yaml):

AWSTemplateFormatVersion: '2010-09-09'
Description: Template for an instnce with Base Amazon Linux AMI + CLI Tools 
Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair
    Default: einsteinish
    Type: String
  AWSAccessKey:
    Type: String
  AWSSecretAccessKey:
    Type: String
  AWSAmiId:
    Default: ami-60b6c60a
    Type: String
  InstanceType:
    Default: t2.micro
    Type: String
    AllowedValues:
      - t2.micro
      - t2.medium
  Bucket:
    Default: bogo-aws
    Description: Bucket for staged assets.
    Type: String
  Prefix:
    Default: autoscaling/
    Description: Prefix for staged assets.
    Type: String
Resources:
  ElasticLoadBalancer:
    Type: AWS::ElasticLoadBalancing::LoadBalancer
    Properties:
      AvailabilityZones: !GetAZs ''
      Listeners:
        - LoadBalancerPort: '80'
          InstancePort: '80'
          Protocol: HTTP
      HealthCheck:
        Target: HTTP:80/
        HealthyThreshold: '3'
        UnhealthyThreshold: '5'
        Interval: '30'
        Timeout: '5'
  Ec2Instance:
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /home/ec2-user/examplefiles-as.zip:
              source: !Sub 'https://s3.amazonaws.com/${Bucket}/${Prefix}static/examplefiles-as.zip'
              group: ec2-user
              mode: '000600'
              owner: ec2-user
            /home/ec2-user/as-bootstrap.sh:
              source: !Sub 'https://s3.amazonaws.com/${Bucket}/${Prefix}static/as-bootstrap.sh'
              group: ec2-user
              mode: '000600'
              owner: ec2-user
          packages:
            yum:
              httpd: []
              php: []
              mysql: []
              php-mysql: []
    Properties:
      Tags:
        - Key: Name
          Value: Instance with Command-Line Tools
      DisableApiTermination: 'true'
      ImageId: !Ref 'AWSAmiId'
      IamInstanceProfile: !Ref 'RootInstanceProfile'
      InstanceType: !Ref 'InstanceType'
      KeyName: !Ref 'KeyName'
      SecurityGroups:
        - !Ref 'Ec2SecurityGroup'
      UserData: !Base64
        Fn::Sub: "#!/bin/bash\n# Get Updates _Before_ CfnInit Runs\nyum update -y\
          \ --security\n# Helper function\nfunction error_exit \n{\n  /opt/aws/bin/cfn-signal\
          \ -e 1 -r \"$1\" \"${WaitHandle}\"\n  exit 1\n}\n# Install packages and\
          \ write files in AWS::CloudFormation::Init\n/opt/aws/bin/cfn-init -s ${AWS::StackName}\
          \ -r Ec2Instance     --region ${AWS::Region} || error_exit 'Failed to run\
          \ cfn-init'\n# Update SSHd Config to listen on port 22 and 80\nsed -i '/^#Port\
          \ 22$/c\\Port 22' /etc/ssh/sshd_config\n# Restart SSHd.\n# Update suders\
          \ file to not require a TTY for sudo.\nsed -i 's/^Defaults    requiretty/#&/'\
          \ /etc/sudoers\n/etc/init.d/sshd restart\nsudo -i -u ec2-user /opt/aws/bin/cfn-describe-stack-resources\
          \ --region ${AWS::Region} --stack-name ${AWS::StackName} --show-long | grep\
          \ -E \"ElasticLoadBalancer|Ec2SecurityGroup|Ec2Instance\" | cut -d ',' -f2,3\
          \ > /home/ec2-user/environments.txt\necho \"ElasticLoadBalancer, `aws elb\
          \ describe-load-balancers --region ${AWS::Region} --query 'LoadBalancerDescriptions[0].LoadBalancerName'\
          \ --output=text`\" >> /home/ec2-user/environments.txt\necho \"AMIId, `curl\
          \ http://169.254.169.254/latest/meta-data/ami-id`\" >> /home/ec2-user/environments.txt\n\
          echo \"KeyName, `curl http://169.254.169.254/latest/meta-data/public-keys/\
          \ | sed 's/0=//g'`\" >> /home/ec2-user/environments.txt\necho \"AvailabilityZone,\
          \ `curl http://169.254.169.254/latest/meta-data/placement/availability-zone`\"\
          \ >> /home/ec2-user/environments.txt\necho \"SecurityGroup, `curl http://169.254.169.254/latest/meta-data/security-groups`\"\
          \ >> /home/ec2-user/environments.txt\nchown ec2-user:ec2-user /home/ec2-user/environments.txt\n\
          # Signal Success to CloudFormation Stack WaitHandle\n/opt/aws/bin/cfn-signal\
          \ -e 0 -r \"cfn-int setup complete\" \"${WaitHandle}\"\n"
    DependsOn: RootInstanceProfile
    Type: AWS::EC2::Instance
  RootRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
  RolePolicies:
    DependsOn: RootRole
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: root
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - ec2:*
              - autoscaling:*
              - sns:*
              - elasticloadbalancing:*
            Resource: '*'
          - Effect: Deny
            Action: autoscaling:CreateLaunchConfiguration
            Resource:
              - '*'
            Condition:
              StringNotEquals:
                autoscaling:InstanceType:
                  - t2.micro
                  - t2.nano
                  - t2.small
                  - t2.medium
          - Effect: Deny
            Action:
              - autoscaling:CreateAutoScalingGroup
              - autoscaling:UpdateAutoScalingGroup
            Resource: '*'
            Condition:
              NumericGreaterThan:
                autoscaling:MaxSize: '5'
          - Action:
              - ec2:RunInstances
            Effect: Deny
            Resource: arn:aws:ec2:*:*:instance/*
            Condition:
              StringNotEquals:
                ec2:InstanceType:
                  - t2.micro
                  - t1.micro
                  - m1.small
          - Action: iam:CreateServiceLinkedRole
            Effect: Allow
            Resource: arn:aws:iam::*:role/aws-service-role/*

          - Action: sts:AssumeRole
            Effect: Allow
            Resource: arn:aws:iam::*:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling
            Condition:
              StringLike:
                iam:AWSServiceName: autoscaling.amazonaws.com
      Roles:
        - !Ref 'RootRole'
  RootInstanceProfile:
    DependsOn: RolePolicies
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref 'RootRole'
  Ec2SecurityGroup:
    Properties:
      GroupDescription: SSH access on port 80 and 22
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          FromPort: '22'
          IpProtocol: tcp
          ToPort: '22'
        - CidrIp: 0.0.0.0/0
          FromPort: '80'
          IpProtocol: tcp
          ToPort: '80'
    Type: AWS::EC2::SecurityGroup
  WaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    Properties:
      Handle: !Ref 'WaitHandle'
      Timeout: '1200'
  WaitHandle:
    Type: AWS::CloudFormation::WaitConditionHandle
Outputs:
  Instance:
    Description: DNS Name of the newly created EC2 instance
    Value: !GetAtt 'Ec2Instance.PublicDnsName'
  AvailabilityZoneName:
    Description: Availability Zone containing your instances
    Value: !GetAtt 'Ec2Instance.AvailabilityZone'
  LoadBalancerName:
    Description: ElasticLoadBalancer.
    Value: !Ref 'ElasticLoadBalancer'
  Ec2SecurityGroupName:
    Description: Ec2SecurityGroup.
    Value: !Ref 'Ec2SecurityGroup'
  Info:
    Description: Outputs for additional information
    Value: !Sub '{"HostDNS" : "${Ec2Instance.PublicDnsName}","InstanceId" : "${Ec2Instance}","Connection"
      : "ec2-user@${Ec2Instance.PublicDnsName}"}'

Let's go into the base instance:

$ ssh -i einsteinish.pem ec2-user@34.207.79.231
The authenticity of host '34.207.79.231 (34.207.79.231)' can't be established.
ECDSA key fingerprint is SHA256:O4ieC7Jw7IduZ3ASjaGG4n9aXSBhk3cfYOTsH9vPGcU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '34.207.79.231' (ECDSA) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/
No packages needed for security; 84 packages available
Run "sudo yum update" to apply all updates.
Amazon Linux version 2018.03 is available.
[ec2-user@ip-172-31-39-200 ~]$ ls
as-bootstrap.sh  environments.txt  examplefiles-as.zip
[ec2-user@ip-172-31-39-200 ~]$

[ec2-user@ip-172-31-39-200 ~]$ cat as-bootstrap.sh 
#!/bin/sh
yum -y install httpd php mysql php-mysql
chkconfig httpd on
/etc/init.d/httpd start
cd /tmp
wget https://s3.amazonaws.com/bogo-aws/autoscaling/static/examplefiles-as.zip
unzip examplefiles-as.zip
mv examplefiles-as/* /var/www/html

[ec2-user@ip-172-31-39-200 ~]$ cat environments.txt 
ElasticLoadBalancer, base-inst-ElasticL-WL90T2TGLEUB
AMIId, ami-60b6c60a
KeyName, einsteinish
AvailabilityZone, us-east-1c
SecurityGroup, base-instance-Ec2SecurityGroup-YIOAG7W926DT

Configure AWS CLI

On the base instance, we may want to configure AWS CLI.

[ec2-user@ip-172-31-39-200 ~]$ aws configure
AWS Access Key ID [None]: 
AWS Secret Access Key [None]: 
Default region name [None]: us-east-1
Default output format [None]:

Creating a launch configuration

A launch configuration defines the instances that are created in response to the increased demand. Autoscaling calls this launch configuration which is a set of parameters describing what kind of instances to launch. Auto Scaling Group then tells the system what to do after the launch.

We'll create the launch configuration via CLI on the SSH shell on the base instance.

aws autoscaling create-launch-configuration \
--image-id <YourAMIIdHere> \
--instance-type t2.micro \
--key-name <YourKeyNameHere> \
--security-groups <YourSecurityGroup> \
--user-data file:///home/ec2-user/as-bootstrap.sh \
--launch-configuration-name <YourLaunchConfigurationName>

The actual command looks like this:

[ec2-user@ip-172-31-39-200 ~]$ aws autoscaling create-launch-configuration --image-id ami-60b6c60a --instance-type t2.micro --key-name einsteinish --security-groups base-instance-Ec2SecurityGroup-YIOAG7W926DT --user-data file:///home/ec2-user/as-bootstrap.sh --launch-configuration-name bogo-asg-launch-configuration
[ec2-user@ip-172-31-39-200 ~]$

Creating an Auto Scaling Group

We'll create an Auto Scaling Group : 1 <= instances <= 4.

We need to understand the Auto Scaling takes time, at least two minutes, for scaling out:

CloudWatch keeps polling for system info, in our case, about the CPU utilization. In general, it takes 1 minute to gather the statistics.
Auto Scaling is also polling on the health of the instances, and it make take up 1 minute.
On top of the two, we need to consider the instance booting time.
Finally, the load balance is running a couple of health-check cycles.

Here is the command for creating ASG:

aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name <YourAutoScalingGroupName> \
--availability-zones <YourAvailabilityZoneHere> \
--launch-configuration-name <YourLaunchConfigurationName> \
--load-balancer-names <YourElasticLoadBalancerHere> \
--max-size 4 --min-size 1

Actual command to use:

[ec2-user@ip-172-31-39-200 ~]$ aws autoscaling create-auto-scaling-group \
> --auto-scaling-group-name bogo-asg \
> --availability-zones us-east-1c \
> --launch-configuration-name bogo-asg-launch-configuration \
> --load-balancer-names base-inst-ElasticL-WL90T2TGLEUB \
> --max-size 4 \
> --min-size 1
[ec2-user@ip-172-31-39-200 ~]$

We can verify the ASG instance is working properly:

We need to also check if ASG re-launches a new instance after a termination of an instance:

This is a minor thing but we can see the instance from ASG has no tag for the name. So, we may also want to give it a name:

[ec2-user@ip-172-31-39-200 ~]$ aws autoscaling create-or-update-tags \
> --tags "ResourceId=bogo-asg, ResourceType=auto-scaling-group, \
> Key=Name, Value=bogo-ASG-Web-Server, PropagateAtLaunch=true"

If we stop the running instance, the ASG will re-launch a named instance:

Autoscaling with Load Balancer

Actually, ASG instances are being added to Load Balancer. This is already done when we create our Auto Scaling Group:

aws autoscaling create-auto-scaling-group \
--auto-scaling-group-name bogo-asg \
--availability-zones us-east-1c \
--launch-configuration-name bogo-asg-launch-configuration \
--load-balancer-names base-inst-ElasticL-WL90T2TGLEUB \
--max-size 4 \
--min-size 1

To verify, we can go to Load Balance on the console:

Or we can use DNS name for LB:

Create Simple Notification Service (SNS) for Autoscaling

In this section, we'll setup notifications from Auto Scaling via SNS when there are ASG scaling in/out activities.

Once subscription to the topic requested, we need to confirm it:

Then, we get confirmation for the subscription from AWS:

We may want to keep the record of the SNS Notification Topic arn: "arn:aws:sns:us-east-1:526262051452:bogo-ASG-Topic"

Now, it's the time for creating Auto Scaling Notification:

[ec2-user@ip-172-31-39-200 ~]$ aws autoscaling put-notification-configuration \
> --auto-scaling-group-name bogo-asg \
> --topic-arn arn:aws:sns:us-east-1:526262051452:bogo-ASG-Topic \
> --notification-types autoscaling:EC2_INSTANCE_LAUNCH autoscaling:EC2_INSTANCE_TERMINATE
[ec2-user@ip-172-31-39-200 ~]$

Create Auto Scaling Policies

In this section, we'll create the Auto Scaling policies using CloudWatch's CPU Utilization metric: scaling up when CPU >= 50%, scaling down when CPU <= 30%.

Let's create the policies.

Scale up policy:

[ec2-user@ip-172-31-39-200 ~]$ aws autoscaling put-scaling-policy \
> --policy-name bogo-scale-up-policy \
> --auto-scaling-group-name bogo-asg \
> --scaling-adjustment 1 \
> --adjustment-type ChangeInCapacity \
> --cooldown 300 \
> --query 'PolicyARN' \
> --output text
arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:d3876e0f-55b5-446b-a486-1657f1f65baa:autoScalingGroupName/bogo-asg:policyName/bogo-scale-up-policy
[ec2-user@ip-172-31-39-200 ~]$

Scale down policy:

[ec2-user@ip-172-31-39-200 ~]$ aws autoscaling put-scaling-policy \
> --policy-name bogo-scale-down-policy \
> --auto-scaling-group-name bogo-asg \
> --scaling-adjustment -1 \
> --adjustment-type ChangeInCapacity \
> --cooldown 300 \
> --query 'PolicyARN' \
> --output text
arn:aws:autoscaling:us-east-1:526262051452:scalingPolicy:0bbd4c59-d53c-4d86-b183-61a1745eabb5:autoScalingGroupName/bogo-asg:policyName/bogo-scale-down-policy

Create CloudWatch High/Low CPU Alerts

In this section, we'll create a CloudWatch alarm to monitor the aggregate average CPU utilization of our ASG instances. It will trigger scale-up/down policies we setup in the previous section.

Click "Select metric".

Do the same shown below. Note that under "Actions" section, we need to click "Delete" on the Notification box, and then click "+AutoScaling Action":

Click "Create Alarm".

Now, we've done for High CPU alarm (CPU >= 50%):

Let's setup the Low CPU alarm (CPU <= 30%) following the similar steps (only some of the steps are shown here).

Initially, we get INSUFFICIENT DATA for the alarm (Low CPU) just setup:

But eventually, it will have appropriate CPU data, and as shown in the picture below, we get Low CPU Alarm because our instance is not using any CPUs: