Chef Knife Bootstrap a Node on EC2 Ubuntu 14.04

Chef

In the previous two chapters, we have setup the Chef server and a workstation online:

1. Chef server install on Ubuntu 14.04
2. Chef workstation setup on EC2 Ubuntu 14.04

So, now we can try to bootstrap a Chef client on a sample node. We will use another Ubuntu instance of EC2.

As we'll see this section is not complete since I could not bootstrap a node while still an instance launched. This is due to the ssh setup, I believe, and will be fixed later.

If anyone can fix my issue, please let me know : contactus@bogotobogo.com.

A node is any physical, virtual, or cloud machine that is configured to be maintained by a chef-client. The knife bootstrap command is a common way to install the chef-client on a node. The default for this approach assumes that node can access the Chef website so that it may download the chef-client package from that location.

The bootstrapping process involves setting up a Chef client on a node. Chef client communicates with the Chef server to receive directions for its own configuration. After the client receives the policy, it applies to the node to ensure the client is configured as per the directions of the server.

This process will simply configure our new instance to be under the control of our Chef management system. We can then configure it however we would like by creating policies on our workstation and uploading them to our server.

The knife ec2 is used to manage API-driven cloud servers that are hosted by Amazon EC2. Chef Knife plugin for EC2 gives knife the ability to create, bootstrap, and manage EC2 instances.

To install the knife ec2 plugin using RubyGems, we want to run the following command:

ubuntu@workstation:~$ sudo apt-get install make g++ 
ubuntu@workstation:~$ sudo apt-get install libxml2 libxml2-dev libxslt1-dev
ubuntu@workstation:~$ sudo apt-get install zlib1g-dev
ubuntu@workstation:~$ sudo /opt/chef/embedded/bin/gem install knife-ec2
...
Successfully installed excon-0.41.0
Successfully installed formatador-0.2.5
Successfully installed net-scp-1.2.1
Successfully installed fog-core-1.25.0
Successfully installed fog-json-1.0.0
Successfully installed mini_portile-0.6.1
Successfully installed nokogiri-1.6.5
Successfully installed inflecto-0.0.2
Successfully installed fog-brightbox-0.7.0
Successfully installed fog-softlayer-0.3.25
Successfully installed fog-1.23.0
Successfully installed gssapi-1.0.3
Successfully installed httpclient-2.5.3.3
Successfully installed rubyntlm-0.1.1
Successfully installed uuidtools-2.1.5
Successfully installed nori-1.1.5
Successfully installed httpi-0.9.7
Successfully installed wasabi-1.0.0
Successfully installed gyoku-1.2.2
Successfully installed akami-1.2.2
Successfully installed savon-0.9.5
Successfully installed little-plugger-1.1.3
Successfully installed logging-1.8.2
Successfully installed winrm-1.2.0
Successfully installed winrm-s-0.2.2
Successfully installed eventmachine-1.0.3
Successfully installed em-winrm-0.6.0
Successfully installed knife-windows-0.8.2
Successfully installed knife-ec2-0.10.0
29 gems installed
...

On the workstation, create the required cookbooks and upload to the Chef Server.

ubuntu@workstation:~/chef-repo$ cd ~/chef-repo/cookbooks/

Clone the repositories from GitHub:

ubuntu@workstation:~/chef-repo/cookbooks$ git clone https://github.com/opscode-cookbooks/apt/
ubuntu@workstation:~/chef-repo/cookbooks$ git clone https://github.com/socrata-cookbooks/java
ubuntu@workstation:~/chef-repo/cookbooks$ git clone https://github.com/opscode-cookbooks/openssl
ubuntu@workstation:~/chef-repo/cookbooks$ git clone https://github.com/opscode-cookbooks/tomcat
ubuntu@workstation:~/chef-repo/cookbooks$ git clone https://github.com/sethvargo/chef-sugar.git

Upload the cookbooks to the server:

ubuntu@workstation:~/chef-repo/cookbooks$ knife cookbook upload java apt
Uploading java           [1.29.0]
Uploading apt            [2.6.0]
Uploaded 2 cookbooks.

ubuntu@workstation:~/chef-repo/cookbooks$ knife cookbook upload chef-sugar openssl tomcat
Uploading chef-sugar     [2.4.1]
Uploading openssl        [2.0.1]
Uploading tomcat         [0.16.2]
Uploaded 3 cookbooks.

We can use to configure roles is JSON.

A role is a way to define certain patterns and processes that exist across nodes in an organization as belonging to a single job function. Each role consists of zero (or more) attributes and a run-list. Each node can have zero (or more) roles assigned to it. When a role is run against a node, the configuration details of that node are compared against the attributes of the role, and then the contents of that role's run-list are applied to the node's configuration details. When a chef-client runs, it merges its own attributes and run-lists with those contained within each assigned role.

We will create a role named "my_app", enter:

$ knife role create my_app

Actually, before we issue the command knife role create, we need to set $EDITOR as below:

export EDITOR=$(which vi)

Then, at the knife role create command, editing window will be opened.

In the editor, fill the file contents in JSON format as below:

{
  "name": "my_app",
  "description": "",
  "json_class": "Chef::Role",
  "default_attributes": {

  },
  "override_attributes": {

  },
  "chef_type": "role",
  "run_list": ["recipe[apt]", "recipe[tomcat]"],
  "env_run_lists": {

  }
}

After finishing edit, we get:

Created role[my_app]

Login to the Chef Server and navigate to the Roles tab, we can see the role we created in the previous section:

Now we want to check if knife ec2 works:

ubuntu@workstation:~/chef-repo/.chef$ knife ec2 server list
ERROR: Fog::Compute::AWS::Error: UnauthorizedOperation => You are not authorized to perform this operation.

We should modify User Policy => Set Permissions as shown below (we're using the default permission setting):

With knife.rb looks like this:

knife[:region] = "us-east-1"
knife[:aws_access_key_id] = "AK*******"
knife[:aws_secret_access_key] = "bm******"

Now we can run it successfully:

ubuntu@workstation:~/chef-repo/.chef$ knife ec2 server list
Instance ID  Name                 Public IP       Private IP     Flavor    Image         SSH Key              Security Groups                      IAM Profile  State  
i-f5b1fa14   Chef_Workstation     54.172.74.156   172.31.48.124  t2.micro  ami-9eaa1cf6  Chef_Workstation     Chef_11_Server                                    running
i-5f7f31be   Chef11_EC2           54.172.41.43    172.31.52.254  t2.micro  ami-9eaa1cf6  Chef_11_Server       Chef11_Server_On_EC2_Security_Group               running

Now we are all set to bootstrap an EC2 instance with the my_app role with the following command:

ubuntu@workstation:~/chef-repo$
sudo knife ec2 server create -I ami-9eaa1cf6 -r "role[my_app]" -Z us-east-1c -g Chef11_Server_On_EC2_Security_Group --ssh-key Chef_11_Server -i Chef_11_Server.pem -f t2.micro -A 'AK***' -K "bm***" --region us-east-1 --ssh-user ubuntu
 
ubuntu@workstation:~/chef-repo$
sudo knife ec2 server create -r "role[my_app]"
 
Instance ID: i-5fedb1be
Flavor: t2.micro
Image: ami-9eaa1cf6
Region: us-east-1
Availability Zone: us-east-1c
Security Group Ids: Chef11_Server_On_EC2_Security_Group
Tags: Name: i-5fedb1be
SSH Key: Chef_11_Server

Waiting for EC2 to create the instance.................
Public DNS Name: ec2-54-174-126-235.compute-1.amazonaws.com
Public IP Address: 54.174.126.235
Private DNS Name: ip-172-31-55-151.ec2.internal
Private IP Address: 172.31.55.151

Waiting for sshd access to become availabledone
Connecting to ec2-54-174-126-235.compute-1.amazonaws.com
Failed to authenticate ubuntu - trying password auth
Enter your password: 
ERROR: Net::SSH::AuthenticationFailed: Authentication failed for user ubuntu@ec2-54-174-126-235.compute-1.amazonaws.com@ec2-54-174-126-235.compute-1.amazonaws.com
ubuntu@workstation:~/chef-repo$ 

There are some args overlapped with the bootstrap args, here is may knife.rb:

log_level                :info
log_location             STDOUT
node_name                'bogo-chef'
client_key               '/home/k/chef-repo/.chef/bogo-chef.pem'
validation_client_name   'chef-validator'
validation_key           '/home/k/chef-repo/.chef/admin.pem'
chef_server_url          'https://ec2-54-172-41-43.compute-1.amazonaws.com'
syntax_check_cache_path  '/home/k/chef-repo/.chef/syntax_check_cache'
cookbook_path [ '~/chef-repo/cookbooks' ]

# EC2 subcommand
knife[:availability_zone] = "us-east-1c"
knife[:region] = "us-east-1"
knife[:image] = "ami-9eaa1cf6"
knife[:flavor] = "t2.micro"

knife[:aws_access_key_id] = "AK*"
knife[:aws_secret_access_key] = "bm*"
knife[:aws_ssh_key_id] = "Chef_11_Server"

The options used in the knife ec2 server create command are as below:

1. -I IMAGE, --image IMAGE
   The name of the image that identifies the operating system (and version) that will be used to create the virtual machine.
2. -r RUN_LIST, --run-list RUN_LIST
   A comma-separated list of roles and/or recipes to be applied.
3. -Z ZONE, --availability-zone ZONE
   The name of the availability zone. Each availability zone is unique within a region. Default: us-east-1b.
4. -g X,Y,Z, --security-group-ids X,Y,Z
   A comma-separated list of security group identifiers. Required when using Amazon Virtual Private Cloud.
5. --ssh-key KEY
   The SSH key for the Amazon EC2 environment. The name of ec2-keypair.
6. -i IDENTITY_FILE, --identity-file IDENTITY_FILE
   The SSH identity file used for authentication. Key-based authentication is recommended. Refers to the private part of that key pair.
7. -f FLAVOR, --flavor FLAVOR
   The name of the flavor that identifies the hardware configuration of the server, including disk space, memory capacity, and CPU priority.
8. -A KEY, --aws-access-key-id KEY
   The access key identifier used with Amazon EC2.
9. -K SECRET, --aws-secret-access-key SECRET
   The secret access key for the API endpoint used with Amazon EC2.
10. --region REGION
    The name of the region from which instances of hosted applications are launched. Each region has a unique endpoint.
11. -x USERNAME, --ssh-user USERNAME
    The SSH user name.

As mentioned earlier, this is what I've got so far, and with some issues to be resolved to finish the bootstrapping a node from EC2 workstation.

Coming soon...

Hi K Hong,

First, I want to say thank you about your great tutorial ablout chef in here: http://www.bogotobogo.com/DevOps/Chef/Chef_Client_Node_Knife_Bootstraping_a_Node_on_EC2_ubuntu_14_04.php
i'm very appreciated.

I follow it, about your issue "could not bootstrap a node while still an instance launched", I bootstrap success, please see the command line in here: 
https://gist.github.com/thienlequang/fc8532e4c86192c5048b

It is about ssh setup, I think you don't have Chef_11_Server.pem  file in your workstation.

Here is my command:
sudo knife ec2 server create -I ami-29ebb519 -r "role[my_app]" -Z us-west-2a -glaunch-wizard-2 --ssh-key thienle-ec2-micro -i thienle-ec2.pem -f t2.micro -A 'AKI***' -K "wz3***" --region us-west-2 --ssh-user ubuntu

I have this pem file "thienle-ec2.pem" this file use for log to my ec2 instance.
Hope it help.

Thank you
Thien