Create multiple EC2 instances rapidly with one shell script

using aws cli

Posted by paul on 2017.10.13

Goal

Use a shell script with embedded awscli command to launch multiple EC2 instances, each with an unique Tags.Name.

Demo

First, asciinema video of the script ec2-creator.sh in action.

  1. The script created 2 EC2 instances (wwwdev1 and wwwdev2) first.
  2. I then ssh into wwwdev1 to verify and log out, ending in Mac Terminal.
  3. The script is used again to launch 2 more EC2 instances (wwwqa11 and wwwqa12).
  4. I then ssh into wwwqa11 to verify and log out, ending in Mac Terminal.
  5. I run "aws ec2 describe-instances ..." command and pull some metadata of EC2 instances that have Tags.Name value that start with "www"
You can easily create 10, 20 or 100 EC2 instances, each with a unique Tags.Name value, by running the script just once. And by using --user-data parameter with AWS CLI command, all EC2 instances have same configs/user-accouts/etc with no manual config required.

Articles on AWS CLI

This is part 2 of a series on AWS CLI tools.

In the first article (Intro to awscli and aws-shell), I covered installing AWS CLI and configuring it to access your AWS account. You need to complete the steps in that 1st article before proceeding.

In this 2nd article, I will discuss how to combine a shell script and AWS CLI commands to automate creating multiple EC2 instances quickly. The shell script will assign an unique Tags.Name value to each EC2. Assigning a unique Tag Name to each EC2 will help with maintaining a clean inventory of EC2 instances.

You should have some experience with launching new EC2 instances in the AWS Console.

If you would like to review the shell script first, the github repo for the script is here: https://github.com/paulcodrpub/ec2creator/blob/master/ec2-creator.sh.

Tools used

  • Mac workstation (or Linux) with AWS CLI installed
  • CentOS 7 for AWS EC2
  • An account with AWS

Details of the shell script

So what exactly does the below command do?

./ec2-creator.sh webserver 1 3

Above script will create 3 EC2 instances with names listed below, each with a unique Tags.Name.

  • webserver1
  • webserver2
  • webserver3

The script requires 3 arguments.

  • webserver: the first part of the hostname. It can be web, db or nodeserver, etc.
  • 1: the starting # of the hostname. It can be 2 or 11 or 22.
  • 3: the total quantity of instances to create. It can be 1 or 12 or 99.

The script will assign an unique Tags.Name to each instance.

The shell script also executes AWS CLI commands to present the list of EC2 instances that were successfully launched.

aws ec2 run-instances

First, let's discuss the AWS CLI command used to launch the EC2 instances.

Below is an example of AWS CLI command to create one new EC2 instance with Tags.Name "webserver1". Note that some of the parameter values used in the command need to be changed for your environment. You can get them after creating a new EC2 using Amazon Console (aka Web GUI).

aws ec2 run-instances \
--image-id ami-f45312121 \
--count 1 \
--instance-type t2.micro \
--key-name aws-sshkey2 \
--security-group-ids sg-5c9121212 \
--block-device-mappings file:///Users/paul/Documents/aws-scripts/mapping.json \
--subnet-id subnet-166121212 \
--user-data file:///Users/paul/Documents/scripts/newcentos-6-7.sh \
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=webserver1}]"

Let's break down the command.

aws ec2 run-instances >>> run-instances actually means create-and-run instance

--image-id ami-f45312121 >>> AMI image name. To follow this tutorial, pick CentOS based AMI.
        Script "newcentos-6-7.sh" (explained below) works with CentOS, not Ubuntu etc.
        *Actual value in your environment will be different.*

--count 1 >>> create 1 instance

--instance-type t2.micro >>> Type of EC2 instance to create

--key-name aws-sshkey2 >>> SSH key generated in AWS Console that you will use
        to ssh into the server. However I actually use my own SSH key added by
        "newcentos-6-7.sh". So I actually don't need this key aws-sshkey2 on EC2.
        But "aws ec2 run-instances" command throws an error if this --key-name value is not
        provided, so I have to include this value.
        *Actual value in your environment will be different.*

--security-group-ids sg-5c9121212 >>> Firewall rule from AWS used for the EC2.
        *Actual value in your environment will be different.*

--block-device-mappings file:///Users/paul/Documents/aws-scripts/mapping.json >>>   For
        setting up the virtual local storage for the EC2.
        *Actual value in your environment will be different.*

--subnet-id subnet-166121212 >>> AWS network the EC2 will be in.
        *Actual value in your environment will be different.*

--user-data file:///Users/paul/Documents/scripts/newcentos-6-7.sh >>>  Shell script
        that will be executed once (and only once) when the EC2 instance
        first boots up.
        *Actual value in your environment will be different.*

--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=webserver1}]" >>> Specifies
        the value of Tags.Name for the EC2.

Note following 2 parameters of the "aws ec2" command uploads two files from the local Mac HD to AWS. These will be explained below.

  • --block-device-mappings
  • --user-data

In order to use "aws ec2" command to create a new EC2 instance, you need to know what parameter values to use. To gather them, you'd need to create an EC2 using the Web GUI.

Create 1 EC2 instance with Web GUI

Create one EC2 instance (let's name it webserver1) using Web GUI so that you can get values for parameters required for AWS CLI command.

When creating the first EC2 instance in Web GUI, use "Advanced | user-data" option and upload shell script "newcentos-6-7.sh". This shell script is discussed in an article I put up earlier: https://paulcodr.co/2017/prep-new-centos-instance/. Using "user-data" and the shell script is highly recommended as you create multiple EC2 instances. It will handle installing commonly used rpms, configuring Ansible client, etc. You shouldn't be doing such tasks manually, one EC2 at a time. You should edit "newcentos-6-7.sh" with your own SSH Public Key, user name, etc.

When creating a new Security Group (aka firewall), use short name (with no blank spaces) for "Security group name". A good example would be "ssh-http-https".

As you create your first EC2 instance, remember to select "Choose an existing key pair" or "Create a new key pair". This would be the last step when creating a new EC2.

Once you are done starting webserver1 in AWS Console, test sshing into the new EC2, webserver1.

Get parameter values

Now you need to get values of the 4 parameters (needed for AWS CLI command) from the new EC2, webserver1. You have 2 options for doing this. First is using "aws ec2" command. Second is (1) ssh into the EC2 and (2) run curl commands.

Get parameter values: Option 1

You will be running "aws ec2" command, which requires AWS CLI is installed. I use virtualenv to keep different python based tools in separate containers. If you do use virtualenv, make sure to activate the one you keep AWS CLI in. This is explained in https://paulcodr.co/2017/awscli-and-aws-shell/.

In the AWS Console, look up the "Instance ID" of webserver1 (ex: i-0222d3322121212). Run following 5 commands in your Terminal to get the 4 parameter values. Copy and paste the commands in so that all characters including comma at end are included. Copy the 4 outputs into a text editor.

# Replace "i-0222d3322121212" with your EC2's
_instid=i-0222d3322121212

# --image-id
echo -e "AMI Image ID:"; aws ec2 describe-instances --instance-ids $_instid | grep -i imageid | awk '{print $2}' | tr -d \" | tr -d ,

# --key-name
echo -e "SSH Key Name:"; aws ec2 describe-key-pairs | grep -i keyname | awk '{print $2}' | tr -d \" | tr -d ,

# --security-group-ids
echo -e "Security Group:"; aws ec2 describe-instances --instance-ids $_instid  | grep SecurityGroups -A10 | grep -i groupid | awk '{print $2}' | tr -d \" | tr -d ,

# --subnet-id
echo -e "SubnetID:"; aws ec2 describe-instances --instance-ids $_instid | grep -i subnetid | awk '{print $2}' | tr -d \" | tr -d ,

Get parameter values: Option 2

Running curl command after sshing into the EC2

SSH into webserver1.

Run following 4 curl commands exactly as shown to get the values for the 4 parameters. When running curl command to get the metadata, trailing "/" does matter. If there is a trailing "/", you should include it.

# 1
curl http://169.254.169.254/latest/meta-data/ami-id

You would use the output exactly as it's shown.

Ex. OUTPUT:  ami-46a1b123        (This is assigned by AWS.)


# 2
curl http://169.254.169.254/latest/meta-data/public-keys/

With this example output, you'd use "aws-sshkey2" as the parameter value.
This output is whatever you picked for the public-keys name.

Ex. OUTPUT:  0=aws-sshkey2


# 3
To run this command you'd run it in 2 stages.
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/

You will see a Mac address (ex:  0e:bc:be:10:5b:7n).
Add that Mac and "security-group-ids" at end. You'd run this command next.
Remember to update the Mac address with your EC2's Mac.
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/0e:bc:be:10:5b:7n/security-group-ids

Ex. OUTPUT:  sg-7643b702


# 4
To run this command you'd run it in 2 stages.
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/

You will see a Mac address (ex:  0e:bc:be:10:5b:7n).
Add that Mac and "subnet-id" at end. You'd run this command next.
Remember to update the Mac address with your EC2's Mac.
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/0e:bc:be:10:5b:7n/subnet-id

EX. OUTPUT:  subnet-e8b790b1   (This is assigned by AWS.)

You now have the 4 parameter values. Store them in a text editor.

2 text files as parameter for awscli command

So you have created an EC2 instance, webserver1, using AWS Console and got values for the 4 parameters. You still need values for 2 more parameters, which are actually 2 text files.

In our AWS CLI example, you will need 2 text files: newcentos-6-7.sh and mapping.json

File 1: newcentos-6-7.sh

Remember to use shell script file "newcentos-6-7.sh" as the value for "user-data". This shell script is discussed in https://paulcodr.co/2017/prep-new-centos-instance/. Using the shell script for "user-data" is highly recommended as you create multiple EC2 instances. Each EC2 instance would get commonly installed rpms, configured Ansible client, add SSH key, and etc, all automatically. You shouldn't be doing such tasks manually, one EC2 at a time.

You should edit "newcentos-6-7.sh" with your own SSH Public Key, user name, etc. You should also update the folder path so that it points to the actule file on your Mac.

File 2: mapping.json

Below parameter specifies that a text file named "mapping.json" will be used to set up the EBS storage.

--block-device-mappings file:///Users/paul/Documents/aws-scripts/mapping.json

You should update the folder path so that it points to the actule file on your Mac.

Without --block-device-mappings parameter, the EC2 instance would have just 8G of EBS storage. AWS allows up to 30G without extra fee, but you have to specify 30G. To assign 30G storage when using "aws ec2" command, you need to include a .json file as a parameter value.

Create mapping.json with following. The file can be stored anywhere on your Mac. I use "/Users/paul/Documents/aws-scripts/mapping.json" in this example.

[
  {
    "DeviceName": "/dev/sda1",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 30,
      "VolumeType": "gp2"
    }
  }
]

With 'DeleteOnTermination' set to true, the storage volume will be deleted automatically when the EC2 instance is deleted. Without this setting, you have to delete the EBS volume manually after deleting the EC2 instances.

Run AWS CLI command to create the 2nd EC2

You now have all 6 parameter values required to run "aws ec2" command to launch a new EC2 instance. We will name this EC2 instance as "webserver2". You can now create and run the "aws ec2" command to create a new EC2 instance. Let's create the command in a text editor.

Prepare the AWS CLI command

  1. Copy following command into a text editor
  2. aws ec2 run-instances \
    --image-id ami-f45312121 \
    --count 1 \
    --instance-type t2.micro \
    --key-name aws-sshkey2 \
    --security-group-ids sg-5c9121212 \
    --block-device-mappings file:///Users/paul/Documents/aws-scripts/mapping.json \
    --subnet-id subnet-166121212 \
    --user-data file:///Users/paul/Documents/scripts/newcentos-6-7.sh \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=webserver1}]"
    
  3. Update following 4 parameter values with the values you got while creating webserver1. These values were saved into a text editor earlier. These values are available in the AWS Console also.
    • --image-id
    • --key-name
    • --security-group-ids
    • --subnet-id
  4. Update value in the last line. This will set the Tags.Name to "webserver2" instead.
  5. BEFORE
    Tags=[{Key=Name,Value=webserver1}]"
    
    AFTER
    Tags=[{Key=Name,Value=webserver2}]"
    
  6. Make sure that following 2 paremeters have correct path/filename to match your Mac's environment.
    • --block-device-mappings
    • --user-data
  7. Now in the text editor, you should see following:
  8. aws ec2 run-instances \
    --image-id ami-f45312121 \
    --count 1 \
    --instance-type t2.micro \
    --key-name aws-sshkey2 \
    --security-group-ids sg-5c9121212 \
    --block-device-mappings file:///Users/paul/Documents/aws-scripts/mapping.json \
    --subnet-id subnet-166121212 \
    --user-data file:///Users/paul/Documents/scripts/newcentos-6-7.sh \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=webserver2}]"
    

Run the AWS CLI command

  1. Open Terminal (or iTerm) on your Mac
  2. If you use virtualenv, activate the virtualenv for AWS CLI. Virtualenv is explained in https://paulcodr.co/2017/awscli-and-aws-shell/.
  3. Copy the aws command from the text editor into the Terminal and run it.
  4. If all is well, you should see an output in Terminal. You should also see the EC2 instance'a name in your AWS Console.
  5. After just a few minutes, you should be able to SSH into the EC2 instance using Public DNS entry visible in the AWS Console.
  6. SSH into the new EC2 and verify the tasks you specified in newcentos-6-7.sh are done.

Automate with a shell script

Now you can create one EC2 instance fairly quickly using the "aws ec2" command. But you do need to update the Tags.Name value for each EC2 for better inventory control of the EC2s.

I do not want to update Tags.Name value manually for each EC2 instance. And I also wanted to be able to create multiple EC2 instances with 1 command, while assining an unique Tags.Name to each EC2. To accomplish these goals, I created a shell script. This shell script works on Mac/Linux with AWS CLI.

The source code for the shell script is on my Github repo.

To run the script

  1. Pull the script ec2-creator.sh from github to your Mac/Linux.
  2. Open ec2-creator.sh and edit the values for the first 7 variables. These were the paremeter values that we discussed earlier in this article.
  3. Make ec2-creator.sh executable.
  4. Run the script as shown in below example. Give starting number and total quantity of instances to create.
  5. ./ec2-creator.sh webserver 3 3
    
  6. You can also review the asciinema video clip shown earlier in this article.

The shell script in above example will create/launch three EC2 instances, each with a unique Tags.Name (webserver3, webserver4, webserver5). And the script will then run a query against the AWS EC2 inventory and list following metadata for each EC2 that has has Tags.Name that starts with "webserver". By this point, there would be 5 EC2 instances in your AWS region.

Tags.Name       InstanceID         PublicDNSName
webserver1	  i-085288d7c12121212	  ec2-12-92-12-1.compute-1.amazonaws.com
webserver2	  i-0a11a079212122222	  ec2-12-12-1-4.compute-1.amazonaws.com
webserver3	  i-0a11a079212133333	  ec2-12-15-8-3.compute-1.amazonaws.com
webserver4	  i-0a11a079212155555	  ec2-12-22-1-5.compute-1.amazonaws.com
webserver5	  i-0a11a079212111111	  ec2-12-19-3-7.compute-1.amazonaws.com

One improvement that I may try to add is alerting the user if running the script will result in having 2 or more EC2 instances with same Tags.Name value.

That is the end and thanks for your time.