Intro to awscli and aws-shell

Posted by paul on 2017.09.10

Length: approximately 1395 words

awscli and aws-shell

In this tutorial I will explain how to install/use awscli and aws-shell. It will end with a shell script that runs an awscli command.

If you are using AWS Console (web GUI) to do anything with AWS other than simply checking status of services, you are doing it wrong.

Most of the changes you make in the web GUI can and should be made with awscli, aka AWS CLI. Also, you can include awscli commands inside of a shell script. When you execute that shell script, the included awscli commands will be executed. This allows you to automate AWS related tasks. However awscli has many different commands and options, and aswcli has no auto-completion feature. Awscli is not very friendly for new users.

And this is where aws-shell comes in. Aws-shell is basically awscli with auto-completion added. Aws-shell is an interactive tool and has auto-completion.

Prerequisites

  • Using Mac as workstation, macOS 10.10.
  • Ability to use vim.
  • Account with AWS.

What will be installed

We will install following in sequence

  • virtualenv
  • awscli
  • aws-shell

Use virtualenv to prep for awscli and aws-shell

On your Mac, install virtualenv first. Virtualenv allows you to set up multiple virtual containers of Python without affecting the default Python environment shipped with your OS. You do not want to modify the default Python environment as modifying it may cause problems for your OS. I highly recommend using virtualenv.

Follow this to install virtualenv on your Mac.
http://sourabhbajaj.com/mac-setup/Python/virtualenv.html

Once you installed virtualenv, create a new environment using any name you prefer. In this tutorial I use 'awsbox'. You can replace "awsbox" with anything else of your choice, such as awsclienv or awscenter.

Run following commands in Terminal to create the virtualenv environment.

mkdir ~/.virtualenvs
cd ~/.virtualenvs
virtualenv awsbox

Note the path to the activate command in ".../awsbox/bin/activate" in this example. The "virtualenv" command basically created folder ~/.virtualenv/awsbox/.

Now activate the newly created environment.

source ~/.virtualenvs/awsbox/bin/activate  

When your virtualenv "awsbox" is activated, you should see following as the shell prompt in your Terminal. Note that "(awsbox)" appears at the very beginning of the prompt. That text shows you've activated a virtualenv environment.

(awsbox)YourComputer:~ username$

You now have virtualen environment named "awsbox" set up. And this means you will need to activate this virtualenv "awsbox" first whenever you run awscli commands or start aws-shell.

To exit the virtualenv, you can type below command. But don't exit the virtualenv yet.

deactivate

Also note while the virtualenv "awsbox" is activated, you can run your regular bash commands such as "ls" or "pwd" or "date".

(awsbox)YourComputer:~ username$ date
Sun Sep 10 11:52:50 PDT 2017

Install awscli

Let's work on the 2nd of the 3 installations. Make sure you are in virtualenv called 'awsbox'. The shell prompt should start with (awsbox).

Install awscli using pip.

pip install awscli --upgrade

Grant awscli access to the AWS

Source: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

From your IAM Console, get your "Access Key ID" and "Secret access key".

In your Mac's Terminal, run following command.

aws configure

Enter the 4 values as prompted. Once done, you should have following 2 files.

~/.aws/credentials
~/.aws/config

Cat those 2 files above and you should see the info you entered with "aws configure". The 2 key values will be different from what you will see in your Terminal.

(awsbox)YourComputer:~ username$ cat ~/.aws/credentials ~/.aws/config
[default]
aws_access_key_id = ABIAI7ABCABCABC123
aws_secret_access_key = EiEOxiqe6jvCWabc123abc123abc123abc123
[default]
output = json
region = us-west-1

Test awscli can access you AWS account

Let's test if you can access your AWS accout with awscli.

You should be in virtualenv "awsbox".

IF the virtualenv is not activated, activate the virtualenv awsbox.

source ~/.virtualenvs/awsbox/bin/activate

Run following command in your Mac's Terminal.

aws ec2 describe-vpcs

Above command pulls info of VPCS from your AWS account. Whenever you start a command with "aws" in your Terminal, you are running an awscli command.

If "aws configure" was completed successfully with correct info, you should see something like following. Below is what I got, with identifiable snippets changed.

{
    "Vpcs": [
        {
            "VpcId": "vpc-bbb11de",
            "InstanceTenancy": "default",
            "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-205a111119",
                    "CidrBlock": "172.32.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],
            "State": "available",
            "DhcpOptionsId": "dopt-e9332aaa",
            "CidrBlock": "172.32.0.0/16",
            "IsDefault": true
        }
    ]
}

Awesome. But surely there are a lot of commands and options to use with awscli. But how can one start using awscli without learning all those commands first?

And this is where aws-shell comes in. Aws-shell is essentially awscli on steroids, with the most useful feature being its auto-completion.

Install aws-shell

This is the last of the 3 installations done in this tutorial.

From: https://github.com/awslabs/aws-shell

Make sure you are in virtualenv called "awsbox". In this virtualenv environment, "awscli" was installed in the 2nd installation.

Install aws-shell with pip.

pip install --upgrade aws-shell

Run first command with aws-shell

Once aws-shell is installed, start aws-shell.

aws-shell

When aws-shell is running, you will see following prompt in your Terminal.

aws>

Note that with aws-shell open, you cannot run normal bash commands such as "ls" or "pwd".

We will try to run "aws ec2 describe-vpcs" command we ran earlier. Note that since aws-shell is running, you will omit "aws". And we will also try out the auto-completion available with aws-shell.

Type in following.

ec2 vpcs

As you type, you will see other possible commands in drop down list. One is "describe-vpcs".

You can use up/down arrow key to highlight the command.

Use down arrow key to highlight "describe-vpcs". Use LEFT arrow key to select the command. Now you can edit the command more or execute it by hitting Enter key.

You should see following again if everything is workin correctly:

  {
      "Vpcs": [
          {
              "VpcId": "vpc-bbb11de",
              "InstanceTenancy": "default",
              "CidrBlockAssociationSet": [
                  {
                      "AssociationId": "vpc-cidr-assoc-205a111119",
                      "CidrBlock": "172.32.0.0/16",
                      "CidrBlockState": {
                          "State": "associated"
                      }
                  }
              ],
              "State": "available",
              "DhcpOptionsId": "dopt-e9332aaa",
              "CidrBlock": "172.32.0.0/16",
              "IsDefault": true
          }
      ]
  }

When you are done with aws-shell, you can quit aws-shell by typing below:

.exit

When you quit aws-shell, you will notice the prompt changes from

#BEFORE

aws>


# AFTER

(awsbox)YourComputer:~ username$

At this point, you are still in virtualenv "awsbox", so you can run awscli commands such as below. You are not using aws-shell, so you need to start any AWS CLI command with "aws". And you can run regular bash commands such as "ls" or "date".

aws ec2 describe-vpcs

Transfer the commands from aws-shell into vim editor

You can copy the commands you ran in aws-shell into a text file easily using .edit command.

Restart aws-shell.

aws-shell

Run following command twice.

ec2 describe-vpcsc2 vpcs

Using ".edit" command in aws-shell, you can seamlessly copy multiple lines of commands from aws-shell OVER TO vim editor. No need to highlight/copy/paste the commands into a text file.

Still in aws-shell, run following command. It will open a text editor, normally "vim" on Mac.

.edit

In the vim editor, you should see the commands you ran in aws-shell. And you should notice that "aws" is automatically added to the beginning of each line. Remember in aws-shell, you did not enter "aws".

aws ec2 describe-vpcs
aws ec2 describe-vpcs

In vim, save the file with a new name and quit vim.

:w ~/Documents/aws-desc.sh
:q!

Remember to specify the filename in vim as you save the file. Note the empty space after :w. In above example, the file is saved under the Documents folder.

Now you are back at aws-shell, which you can tell because of following shell prompt

aws>

Quit aws-shell with following command

.exit

Exit from "awsbox" virtualenv.

deactivate

Use awscli commands in a shell script.

Let's create a shell script that will run the awscli commands.

Open ~/Documents/aws-desc.sh with vim to edit. You should see following 2 lines:

aws ec2 describe-vpcs
aws ec2 describe-vpcs

Let's keep only one aws command.

To turn aws-desc.sh into a working shell script file, edit aws-desc.sh so that it looks like below.

#!/bin/bash
source ~/.virtualenvs/awsbox/bin/activate

aws ec2 describe-vpcs

deactivate

Now, ~/Documents/aws-desc.sh does following

  • turns on the virtualenv awsbox
  • runs the aws command
  • deactivates the virtualenv.

Let's set execute permission and run it

chmod 750 ~/Documents/aws-desc.sh
~/Documents/aws-desc.sh

When you execute shell script, aws-desc.sh, you should see the same output you saw earlier when you ran "aws ec2 describe-vpcs".

{
    "Vpcs": [
        {
            "VpcId": "vpc-bbb11de",
            "InstanceTenancy": "default",
            "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-205a111119",
                    "CidrBlock": "172.32.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],
            "State": "available",
            "DhcpOptionsId": "dopt-e9332aaa",
            "CidrBlock": "172.32.0.0/16",
            "IsDefault": true
        }
    ]
}

Basic workflow with awscli/aws-shell:

So here are the basic workflow I follow to work with awscli.

  • Use aws-shell to test new AWS CLI commands to use. Auto completion in aws-shell will help speed up testing new commands.
  • Next transfer the commands to a shell script to be used as awscli commands.
  • Execute the shell script which runs the awscli commands.

END