Use Vagrant to learn Ansible

Posted by paul on 2017.04.01

Vagrant and Ansible

In my earlier blog, I discussed setting up Vagrant with fully functioning LAN connection. In this blog, I will discuss how to start up multiple Vagrant VMs with Ansible clients enabled automatically. This will help you a lot if you are trying to learn Ansible.

Basically I will use Vagrant's Shell Provisioner which lets you run a shell script once in the initial boot up of a Vagrant VM. The shell script can do anything, including configuring Ansible client on the VM. By using Vagrant, Shell Provisioner, and a Shell script, you can have multiple VMs ready with Ansible client in a matter of minutes. This is great for practicing with Ansible, which is why I ended up learning Vagrant first when I was learning about Ansible.

This tutorial will work best if you are home or in an office LAN with known, unused IPs. You probably won't be able to do the exercises using WiFi in a coffee shop.

Moving parts and tools

  1. macOS computer
  2. Text editor (ex: vim in CentOS. Atom/Sublime in Mac)
  3. Vagrant on your Mac
  4. Multiple Vagrant VMs running CentOS 7
  5. IPs for VMs. I use 192.168.11.211 - 192.168.11.216 here. You should use valid IPs for your LAN.
  6. Multiple Terminal windows on Mac. There is a lot of switching between different Terminal Windows as you are working with Mac prompt, Vagrant, different Vagrant VMs, etc. In order to prevent confusion, you should start the tutorial with 3 Terminal Windows open. I will specify which Terminal to use as we go through the exercises.

Summary of tasks

  1. Set up virtualenv on Mac.
  2. Suspend host key verification
  3. Install Ansible controller on your Mac.
  4. Why use Vagrant for studying Ansible?
  5. Start up two CentOS 7 instances (web1.vag.lan & web2.vag.lan) for test
  6. Modify web1.vag.lan
  7. Modify web2.vag.lan
  8. Manually add Ansible client to web1.vag.lan
  9. Test Ansible controller against web1.vag.lan
  10. We will script Ansible client setup and test it with web2.vag.lan
  11. Update ~/codes/ansible/hosts to manage web2.vag.lan
  12. Vagrant, Shell Provisioner, shell script, and Ansible
  13. Test Ansible with web3.vag.lan
  14. Test Ansible with 3 web servers simultaneously

Set up virtualenv on Mac.

This part is optional, but I highly recommend setting up a separate environment in Python for different uses. I use a separate environment for Ansible controller.

  1. Switch to Terminal03.
  2. Set up virtualenv on your Mac by following the tutorial below.
  3. http://sourabhbajaj.com/mac-setup/Python/virtualenv.html
  4. Set up a Python environment dedicated for Ansible. I named mine "ansible".
  5. When I want to run Ansible command on my Mac, I run following command to enter virtualenv for Ansible.
  6. source ~/.virtualenvs/ansible/bin/activate
  7. It seems a little bit of hassle but I think having the ability to keep separate, clean environments is worth it. I use TextExander to keep the command handy. I just type '#startansible' to enter the virtualenv.

Suspend host key verification

In macOS (and other Unix OSes) when you connect with another Unix like computer using SSH for the first time, a unique identifier is saved in your Mac's file: ~/.ssh/known_hosts. When working with Vagrant VMs, you will be recycling same IP with new OS instances often. To avoid having to edit ~/.ssh/known_hosts every time an IP is reused with a new VM, create ~/.ssh/config and add following lines.

Host web*.vag.lan
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null

It turns off host key verification for all web*.vag.lan hosts. Since we will use web1.vag.lan, web2.vag.lan, etc hostnames for this exercise, we can safely disable host key checking.

You can set Ansible itself to not check host key, but I believe it's all or nothing, which is not secure.

Ansible controller is not very adept at dealing with duplicate SSH keys so I highly recommend suspending host key verification, at least for the VMs.

Install Ansible controller on your Mac.

Summary of installing Ansible controller on a Mac

  1. Activate the virtualenv created for Ansible
  2. Install Ansible on your Mac
  3. Create ~/.ansible.cfg in your Mac
  4. Ansible inventory file
  5. Create a ssh key pair file named "ansible_mac"

Activate the virtualenv created for Ansible

Switch to Terminal03. If you are not using virtualenv, you can skip the part of activating virtualenv.

In your Mac's Terminal03, enter the command to activate the virtualenv created for ansible. Below is an example. The path on your Mac may be different.

source ~/.virtualenvs/ansible/bin/activate

Install Ansible on your Mac

In Terminal03 on your Mac (in virtualenv for Ansible), install Ansible controller by following tutorial below:

http://docs.ansible.com/ansible/intro_installation.html#latest-releases-on-mac-osx

Create ~/.ansible.cfg in your Mac

In your Mac's Terminal03, create ~/.ansible.cfg and paste in following. You will need to modify some values to match your environment.

[defaults]
remote_user = ansible
ansible_ssh_private_key_file = /Users/paul/.ssh/ansible_mac
inventory = ~/codes/ansible/hosts

[privilege_escalation]
become = yes
become_method = sudo
  • Value of remote_user: I prefer using dedicated user account (ex: ansible) on remote servers for use by Ansible, instead of using root.
  • Value of ansible_ssh_private_key_file: I use a dedicated SSH key pair, ansible_mac and ansible_mac.pub, for use by Ansible. This way, the private key file can be shared securely with fellow team members if necessary.
  • Value of inventory: inventory shows the default Ansible hosts file. This file could be named anything and saved anywhere. Ansible uses list of servers in this file to reach the Ansible clients.
  • The [privilege_escalation] section is required because we use non-root user for Ansible on remote Ansible clients.

Ansible inventory file

  1. Ansible controller knows what clients to control by checking an "Ansible hosts file". The default file is ~/codes/ansible/hosts. I will use this default hosts file initially. However I personally do not use the default host inventory file for production use. Instead I have a separate file for each group of servers, based on project, purpose, etc. This helps prevent modifying wrong servers with wrong command. This is critical when you can modify dozens or hundreds of servers with 1 Ansible command.
  2. The inventory file would have following. First section is required in all Ansible hosts file.
  3. [defaults]
    [all:vars]
    remote_user = ansible
    ansible_ssh_private_key_file = /Users/paul/.ssh/ansible_mac
    
    [webservers]
    192.168.11.211
    
  4. The first VM in [webservers] section is the VM we will use with Ansible. Its hostname is web1.vag.lan but for now I will use the IP of the server to connect. And I use "vag" as it's a Vagrant VM. I use "lan" (as for LOCAL AREA NETWORK) in the hostname to show the VM is on the LAN.
  5. The value for ansible_ssh_private_key_file is the SSH private key you are using with Ansible. The SSH key pair is created below.

Create a ssh key pair file named "ansible_mac"

The last step of installing Ansible controller is creating an ssh key pair on your Mac. You could use the ssh key that you may already have on your Mac. However I strongly recommend creating a separate, dedicated SSH key for use with Ansible controller. That way, you can give out the private SSH key to another DevOps engineer on your team to manage the same sets of servers. Or you could have each DevOps engineer create a dedicated SSH key and add the public key to all servers. The key point is you should have a separate, dedicated SSH key for use with Ansible for more granular access control.

On your Mac, create SSH key pair for Ansible controller to use to communicate with Ansible clients. I leave SSH Passphrase blank.

ssh-keygen -t rsa -f ~/.ssh/ansible_mac

You will end up with 2 files in ~/.ssh/: ansible_mac & ansible_mac.pub.

That's it for installing Ansible controller on a Mac. At this point, Terminal03 on your Mac has virtualenv activated for Ansible controller.

Why use Vagrant for studying Ansible?

We will use Vagrant for providing virtual machines (aka VM) to test Ansible with. Now, you may be tempted to skip setting up Vagrant and simply use Amazon AWS or DigitalOcean for practicing Ansible. I personally would recommend against this. Using Vagrant will save you a lot of time when learning Ansible.

To install Vagrant on you Mac, install VirtualBox and then Vagrant on your Mac. If you need more guidance, please check my blog on setting up Vagrant.

Let's set up two new CentOS 7 instances in Vagrant. We are creating 2 Vagrant VM instances for lab with Ansible because both need some manual configuration. For sake of efficiency let's configure both together. Each Vagrant VM will need a Bridged IP for this tutorial. If you are not sure how to do that, please check my other blog.

Start up two CentOS 7 instances (web1.vag.lan & web2.vag.lan) for test

Switch to Terminal01 on your Mac. First create a folder for Vagrant VM web1.vag.lan. I keep all my Git checked in files in ~/codes/, and I keep Vagrant config files in ~/codes/vagrants/. So I will create folder web1 for VM web1.

mkdir ~/codes/vagrants/web1/.
 

Vagrant VM is controlled by a file called Vagrantfile. For web1.vag.lan, create file ~/codes/vagrants/web1/Vagrantfile and add following code to the file. It's very important that you use a valid, free IP address of your LAN for this VM. Do not blindly copy the IP from this example below.


# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # General Vagrant VM configuration.
  config.vm.box = "centos/7"
  config.ssh.insert_key = true
  config.vm.synced_folder ".", "/vagrant", disabled: true
  config.vm.provider :virtualbox do |v|
    v.memory = 512
    v.linked_clone = true
  end

  # Server 1.
  config.vm.define "web1.vag.lan" do |web|
    web.vm.hostname = "web1.vag.lan"
    web.vm.network :public_network, ip: "192.168.11.211", bridge: "en0: Wi-Fi (Airport)", bootproto: "static", gateway: "192.168.11.1"
  end
end

Above Vagrantfile will start CentOS 7 VM named web1.vag.lan, with bridged IP 192.168.11.211.

Still in Mac's Terminal01, create Vagrantfile for web2.vag.lan. We can simply copy the folder like below: * Note no trailing "/"

cp -r ~/codes/vagrants/web1 ~/codes/vagrants/web2

Now you should have ```~/codes/vagrants/web2/Vagrantfile```. Let's edit ```~/codes/vagrants/web2/Vagrantfile``` by making changes in 3 places. Basically we are changing hostname in 2 places, and changing the IP to 192.168.11.212. Make sure to use valid IP and gateway of YOUR LAN.

# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # General Vagrant VM configuration.
  config.vm.box = "centos/7"
  config.ssh.insert_key = true
  config.vm.synced_folder ".", "/vagrant", disabled: true
  config.vm.provider :virtualbox do |v|
    v.memory = 512
    v.linked_clone = true
  end

  # Server 1.
  config.vm.define "web2.vag.lan" do |web|
    web.vm.hostname = "web2.vag.lan"
    web.vm.network :public_network, ip: "192.168.11.212", bridge: "en0: Wi-Fi (Airport)", bootproto: "static", gateway: "192.168.11.1"
  end
end

Both VMs are ready to be started. We will fire up each VM and modify it to allow sshing in as a user.

Modify web1.vag.lan

  1. Check you are at your Mac's Terminal01, which should be at the macOS prompt. We will start up web1.vag.lan, ssh in, and become root of web1.vag.lan.
  2. Change directory to ```~/codes/vagrants/web1``` so that your pwd is ~/codes/vagrants/web1/.
  3. Run vagrant up to fire up web1.vag.lan. Because of Vagrant plugin "vagrant-hostsupdater", you will be prompted to enter root password of YOUR Mac about half way through the boot up. The root password of your Mac is normally the password you use to log into your Mac. This will allow Vagrant to update /private/etc/hosts.
  4. Once web1.vag.lan is up, ssh in with "vagrant ssh". This is effectively sshing into web1.vag.lan as user "vagrant". Become root with "sudo su -". It would look like below. Note line 1's MacBook: means you are logged into your Mac. Line 2's [[email protected] ~]$ means you are logged into web1, as user vagrant. Line 3's [[email protected] ~]# means you are root on web1.
  5. MacBook:web1 paul$ vagrant ssh
    [[email protected] ~]$ sudo su -
    [[email protected] ~]#
    
  6. You are in Terminal01, and logged into CentOS 7 VM, web1.vag.lan.
  7. Run "ip a" and you will notice eth1 is there, but no IP is assigned.
  8. Run "ifup eth1" to bring up the virtual NIC with 192.168.11.211. Eth1 is configured by Vagrantfile with a staic IP, but it doesn't come up by itself after initial bootup. So you need to issue "ifup eth1" command.
  9. Now VM web1.vag.lan is running with a Bridged IP (192.168.11.211).
  10. The official CentOS 7 Vagrant image allows ssh login only via SSH key. We will need to allow sshing in using password. We will first install vim to test network. Additionally, vim has color coding, which helps.
  11. [[email protected] ~]# yum -y install vim
    ...
    ...
    
    Complete!
    
  12. Edit /etc/ssh/sshd_config and change following line. It's line 79.
  13. # BEFORE
    PasswordAuthentication no
    
    # AFTER
    PasswordAuthentication yes
    
  14. Save /etc/ssh/sshd_config and restart sshd service of web1.vag.lan.
  15. [[email protected] ~]# systemctl restart sshd
    
  16. Now create a user that you can use when scping in a file to web1.vag.lan. I'm using "codr" as the username. The 3rd command is setting a password for "codr".
  17. useradd codr
    usermod -aG wheel codr
    passwd codr
    
  18. To give the newly created user root privilege, run "visudo" and uncomment following line. You could directly edit /etc/sudoers also, but not recommended for less experienced. This gives users "codr" and "ansible" sudo privilege.
  19. 
    BEFORE
    # %wheel        ALL=(ALL)       NOPASSWD: ALL
    
    AFTER
    %wheel        ALL=(ALL)       NOPASSWD: ALL
    
  20. Run "ip a", and you will see eth1 is not up. If so, run "ifup eth1" to bring up the NIC with IP 192.168.11.211.
  21. Switch to Terminal02. Test sshing into 192.168.11.211 as user "codr" (or any other username you picked). You will still be asked to verify host key because we suspended verifcation for web*.vag.lan, not 192.168.11.211.
  22. 
    MacBook:~ paul$ ssh [email protected]
    The authenticity of host '192.168.11.211 (192.168.11.211)' can't be established.
    RSA key fingerprint is 1e:74:ec:84:02:59:0e:4e:8c:6e:e4:25:f6:c9:71:f0.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '192.168.11.211' (RSA) to the list of known hosts.
    [email protected]'s password:
    [[email protected] ~]$
    
  23. Once logged in to 192.168.11.211, test "sudo su -" and you should become root on 192.168.11.211 (aka web1.vag.lan).
  24. In Terminal02, log out from 192.168.11.211. Because you logged in as user "codr" and became root, you will need to issue "exit" command twice. You should be back to macOS shell prompt.
  25. Switch to Terminal01, log out from 192.168.11.211. Because you logged in as user "vagrant" and became root, you will need to issue "exit" command twice. You should be back to macOS shell prompt.

Modify web2.vag.lan

  1. On your Mac, check you are in Terminal01. And in shell prompt of macOS.
  2. Change directory to ~/codes/vagrants/web2 to work on web2.vag.lan VM.
  3. Run vagrant up to fire up web2.vag.lan. Because of Vagrant plugin "vagrant-hostsupdater", you will be prompted to enter root password of YOUR Mac. This will allow Vagrant to update /private/etc/hosts on your Mac.
  4. When web2.vag.lan is up, ssh into web2.vag.lan by running vagrant ssh.
  5. Once logged in as vagrant, become root of web2.vag.lan by running "sudo su -".
  6. We will modify web2.vag.lan the same way as web1.vag.lan was modified.
  7. In web2.vag.lan, we will first install "vim" editor.
  8. [[email protected] ~]# yum -y install vim
    ...
    ...
    
    Complete!
    
  9. In web2.vag.lan, edit /etc/ssh/sshd_config and change following line. It's line 79.
  10. # BEFORE
    PasswordAuthentication no
    
    # AFTER
    PasswordAuthentication yes
  11. Save /etc/ssh/sshd_config and restart sshd service of web2.vag.lan.
  12. [[email protected] ~]# systemctl restart sshd
    
  13. Now create a user that you can use to scp in a file to web2.vag.lan. I'm using "codr" as the username. Last of the 3 commands is setting a password for "codr".
  14. useradd codr
    usermod -aG wheel codr
    passwd codr
  15. To give the newly created user root privilege, run "visudo" and uncomment following line. You could directly edit /etc/sudoers also, but not recommended for less experienced. This gives users "codr" and "ansible" sudo privilege.
  16. BEFORE
    # %wheel        ALL=(ALL)       NOPASSWD: ALL
    
    AFTER
    %wheel        ALL=(ALL)       NOPASSWD: ALL
  17. Run "ip a", and you will see eth1 is not up. If so, run "ifup eth1" to bring up the NIC with IP 192.168.11.212.
  18. Switch to Terminal02. SSH into web2.vag.lan as user "codr" (or any other username you picked).
  19. Once logged in to web2.vag.lan, test "sudo su -" and you should become root.
  20. In Terminal02, log out from web2.vag.lan. Because you logged in as user "codr" and became root, you will need to issue "exit" command twice. You should be back to macOS shell prompt.
  21. Switch to Terminal01, log out from web2.vag.lan. Because you logged in as user "vagrant" and became root, you will need to issue "exit" command twice. You should be back to macOS shell prompt.

Manually add Ansible client to web1.vag.lan

We have 2 Vagrant VMs (web1 and web2), each with a Bridged IP.

  1. You will manually configure web1.vag.lan (192.168.11.211) to be an Ansible client. We will do it by hand first in order to understand what's happening under the hood. After web1, we will automate the procedure with a shell script and test it with web2.vag.lan.
  2. Next few steps are all done within web1.vag.lan.
  3. Switch to Terminal01 and check you are in macOS prompt.
  4. You can be in any directory on your Mac, although I recommend ~/codes/vagrants/web1/ to keep things consistent.
  5. Try sshing into web1.vag.lan as user "codr", or whatever username you created earlier.
  6. ssh [email protected]
    
  7. You should see following.
  8. MacBook:web1 codr$ ssh [email protected]
    [email protected]'s password:
    [[email protected] ~]$
    
  9. After logging in, run "sudo su -" so that user "codr" gains root privilege on 192.168.11.211.
  10. Add user "ansible" and add it to group "wheel". You can use any username for Ansible to use, but I use ansible.
  11. useradd ansible
    usermod -aG wheel ansible
  12. Switch to Terminal02 on your Mac. Using scp, copy ansible_mac.pub from YOUR Mac to web1.vag.lan.
  13. scp ~/.ssh/ansible_mac.pub [email protected]:~/
    
  14. You should see following.
  15. PCs-MacBook-Pro:~ codr$ scp ~/.ssh/ansible_mac.pub [email protected]:~/
    [email protected]'s password:
    ansible_mac.pub                                                          100%  403     0.4KB/s   00:00
    PCs-MacBook-Pro:~ codr$
    

    Note ansible_mac.pub from your macOS was copied to /home/codr/ansible_mac.pub on 192.168.11.211.

    Switch to Terminal01 of 192.168.11.211, and run following commands as user "root" in web1.vag.lan. Paste in 1 line at a time to execute. Do not skip any. It configures user "ansible" on web1.vag.lan to accept ssh login with SSH key. And this in turn allows Ansible controller (on your Mac) to communicate with Ansible client on 192.168.11.211 (aka web1.vag.lan).

    mv /home/codr/ansible_mac.pub /home/ansible/
    
    cd /home/ansible/
    
    _username=`pwd | cut -d'/' -f3`
    # Above command assigns "ansible" to variable, _username
    
    mkdir /home/ansible/.ssh
    
    cat ansible_mac.pub >> /home/ansible/.ssh/authorized_keys
    
    chown -R $_username: .ssh
    
    chmod 700 .ssh/
    
    chmod 600 .ssh/authorized_keys
    
  16. Switch to Terminal02 on your Mac, and it should be at macOS prompt. From your Mac, test sshing into 192.168.11.211 as user "ansible", using the SSH key (ansible_mac). When the prompt changes like line 2, it means success.
  17. Macbook:web1 codr$ ssh [email protected] -i ~/.ssh/ansible_mac
    [[email protected] ~]$
    
  18. You are in Terminal02, but sshed into 192.168.11.211. After sshing in as "ansible" into 192.168.11.211, run "sudo su -". User ansible should become root.
  19. Epel-release rpm enables EPEL repo, which has additional RPMs not found in the standard CentOS 7 rpm repository.
  20. As root on 192.168.11.211, run following to install epel-release and ansible rpms. Normally you can install multiple rpms with 1 command such as "yum install vim httpd". But with epel-release and ansible rpms, you need to complete installing epel-release before installing ansible. Using && works.
  21. yum install -y epel-release && yum install -y ansible
    
  22. Verify ansible rpm is installed.
  23. [[email protected] ~]# rpm -qa | grep ansible
    ansible-2.2.1.0-1.el7.noarch
  24. At this point Terminal02 is actually used for the CentOS 7 VM, with IP 192.168.11.211. If you ran a command in Terminal02 now, it would be actually executed inside of the Vagrant VM, NOT your macOS. When working with Vagrant VMs, you need to keep track of what OS instance your Terminal window is using. You can do so by looking at the shell prompt. If not careful, you could delete/modify/move data on wrong servers, something you want to avoid.

Test Ansible controller against web1.vag.lan

Switch to Terminal03 (last used to install Ansible controller) on your Mac. The shell prompt may look like below. "(ansible)" means it's in virtualenv named "ansible".

(ansible)MacBook:~ paul$

Update ~/codes/ansible/hosts

  1. Before doing anything with Ansible, you need to update Ansible hosts file (~/codes/ansible/hosts) on your Mac with correct list of Ansible clients you intend to control.
  2. In your Mac's Terminal03, edit ~/codes/ansible/hosts so it has following code. The only change is the IP address. Ansible hosts file can use hostnames and IP addresses. For a given server, you should pick hostname or IP, not both.
  3. [all:vars]
    remote_user = ansible
    ansible_ssh_private_key_file = /Users/paul/.ssh/ansible_mac
    
    [webservers]
    192.168.11.211
    
  4. Save ~/codes/ansible/hosts.

Run ansible ping command against web1.vag.lan

Now everything is ready.

  • Ansible controller is installed on your Mac.
  • Ansible client is installed on the CentOS 7 VM with IP 192.168.11.211.
  • Your Mac can ssh into the CentOS 7 VM using SSH key ansible_mac.
  • Ansible hosts file (~/codes/ansible/hosts) on your Mac has IP 192.168.11.211.
  • The virtualenv for Ansible is enabled on your Mac.

Make sure you are in Terminal03 of your Mac.

  1. In Terminal03 on your Mac, run following command as a simple test to see if Ansible controller can connect with Ansible client on 192.168.11.211.
  2. In below command, Ansible controller tries to contact all hosts listed in ~/codes/ansible/hosts on your Mac. It's much like ping command used to see if an IP is used by a host. At this state, there's only one Ansible client.
  3. ansible all -m ping -u ansible
    
  4. Let's break down the ansible command.
    • all: This is "all" group from the 1st line in ~/codes/ansible/hosts, meaning all hosts listed in ~/codes/ansible/hosts.
    • -m: to indicate a module is being called.
    • ping: the name of the Ansible module being used.
    • -u: Ansible uses this username to communicate with the Ansible client.
    • ansible: this is the username we created on web1.vag.lan earlier.
  5. If everything is done correctly, you should see following output:
  6. (ansible)MacBook:~ paul$ ansible all -m ping -u ansible
    192.168.11.211 | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    
  7. The text you want to see is "SUCCESS".
  8. Ansible client replied (pong) but nothing was changed on it (false).
  9. Because remote_user is specified in ~/.ansible.cfg and ~/codes/ansible/hosts, you can also run the ping command without "-u ansible".
  10. (ansible)MacBook:~ paul$ ansible all -m ping
    192.168.11.211 | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    
  11. I normally run ansible command without -u key.

Run ansible ad-hoc command against web1.vag.lan

Let's try one more command with Ansible. With Ansible, you can run what's called ad-hoc command, allowing you to run "shell" command on Ansible clients. Let's say you had to verify time on all your servers are correct. Without Ansible, you would have to run a shell command like "date" on multiple Linux servers. You would have to ssh into each server, run "date", and somehow get the output into 1 place (ex: text file) for overall analysis. That's not scaleable. With Ansible, you can run following command once on your Mac and get desired output from multiple remote servers quickly in 1 place.

Run following command in Terminal03 window.

ansible webservers -a "date"

Note above command controls just the servers under "webservers" group in ~/codes/ansible/hosts, not all. But in our example, there's only one server in the hosts file, ~/codes/ansible/hosts, which is 192.168.11.211.

Here's what you should see. The last line is output from the ad-hoc command "hostname".

(ansible)MacBook:~ paul$ ansible webservers -a "date"
192.168.11.211 | SUCCESS | rc=0 >>
Wed Mar 29 18:32:55 UTC 2017

That's the end with 192.168.11.211.

Let's delete web1.vag.lan.

  1. Switch Terminal01. Log out of VM, back to macOS prompt.
  2. Check you are in directory ~/codes/vagrants/web1/
  3. Run "vagrant destroy -f" to delete VM web1.vag.lan.
  4. Enter root password of your Mac so /private/etc/hosts can be updated.

We will script Ansible client setup and test it with web2.vag.lan

Next, we will automate the procedure for installing Ansible client by creating a shell script, setup1.sh. We will test it with web2.vag.lan. Setup1.sh will be created and saved on Mac. But it will be copied to web2.vag.lan and executed in that VM. One thing to remember while creating the shell script is that it should not prompt for any manual input while running. The reason for this is because the shell script will be executed in the Vagrant VM during the initial boot-up, and you do not get a chance to supply any input.

Check you are in Terminal01. If Terminal01 is in a Vagrant VM, log out until you are at macOS Terminal.

Using any text editor of your choice (vim, Atom, Sublime, etc), create shell script (setup1.sh) with following and save it on your Mac. You can save this shell script anywhere on your Mac, as long as it stays there. If possible, you should check the file into Git repository like bitbucket.org. I keep all my Git checked-in files under ~/codes/ on my Mac. And I keep shell scripts in ~/codes/scripts/. So in this tutorial, the shell script is saved on the Mac at ~/codes/scripts/setup1.sh.

Note the actual script I use for myself has a lot more stuff happening, but for clarity, I am including only required parts in the example shell script, setup1.sh.

#!/bin/bash
_user_ansible="ansible"

# bring up eth1
ifup eth1

# install epel-release and few other RPMs that were not included in the Vagrant image
yum -y install epel-release vim

# Uncomment in /etc/sudoers so that group "wheel" can gain sudo access with password
sed -i '/NOPASSWD/a %wheel\      ALL=(ALL)\      NOPASSWD:\ ALL' /etc/sudoers && echo "Granted to wheel group sudo root privilege."

# Install ansible rpm
yum install -y ansible

# Add user ansible
useradd ${_user_ansible} 2> /dev/null
usermod -aG wheel ${_user_ansible} 2> /dev/null
mkdir /home/${_user_ansible}/.ssh 2> /dev/null
touch /home/${_user_ansible}/.ssh/authorized_keys 2> /dev/null

# Append public key of ansible_mac.pub to authorized_keys.
## YOU will NEED to provide your OWN string from YOUR Mac. ##
#############################################################
echo "ssh-rsa  AAAAB3NzaC1yc2EAAAADAQABAAABAQDFcFcKOi8 [email protected]" >> /home/${_user_ansible}/.ssh/authorized_keys

chmod 755 /home/${_user_ansible}/.ssh
chmod 644 /home/${_user_ansible}/.ssh/authorized_keys
chown -R ${_user_ansible}: /home/${_user_ansible}/.ssh

Under "# Append public key of ansible_mac.pub to authorized_keys ...", line 23, you will see echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFcFcKOi8 [email protected]"

All of the text between the double quotes need to be replaced with the content of ~/.ssh/ansible_mac.pub on your Mac.

After saving the setup1.sh on your Mac, use scp command to copy it to web2.vag.lan.

scp ~/codes/scripts/setup1.sh [email protected]:~/

Switch to Terminal02.

If you are still logged into a VM, exit to macOS prompt. From macOS prompt, ssh into web2.vag.lan as user "codr". Provide password for codr when prompted.

Now you are logged into web2.vag.lan as codr.

On web2.lan.vag, run following commands to execute setup1.sh. Running this shell script will configure all necessary steps required for installing Ansible client on web2.vag.lan.

sudo su -

mv /home/codr/setup1.sh /root/.

chown root: /root/setup1.sh

chmod 700 /root/setup1.sh

/root/setup1.sh

If you see following output at the end, it means setup1.sh worked as intended and Ansible client was added to web2.

...
python2-ecdsa.noarch 0:0.13-4.el7                              python2-paramiko.noarch 0:1.16.1-2.el7
python2-pyasn1.noarch 0:0.1.9-7.el7                            sshpass.x86_64 0:1.06-1.el7

Complete!
[[email protected] ~]#

Update ~/codes/ansible/hosts to manage web2.vag.lan

Before testing Ansible client, you need to update ~/codes/ansible/hosts on the Mac.

On your Mac's Terminal03, edit ~/codes/ansible/hosts so it has following code. We will use hostname this time. Only change is the last line.

[all:vars]
remote_user = ansible
ansible_ssh_private_key_file = /Users/paul/.ssh/ansible_mac

[webservers]
web2.vag.lan

In Terminal03, try running ansible ping command on your Mac. If everything is done correctly including setup1.sh, you should see following output:

(ansible)MacBook:~ paul$ ansible all -m ping
web2.vag.lan | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

If it didn't work, go back and troubleshoot. One troubleshooting you should do is testing sshing in from Mac to web2, using ansible_mac ssh key.

ssh [email protected] -i ~/.ssh/ansible_mac

Log out from any VM from both Terminal01 and Terminal02.

Let's delete web2.lan.vag.

  1. In Terminal01, in Mac's shell prompt, change directory to ~/codes/vagrants/web2/
  2. Run "vagrant destroy -f" to delete VM web2.vag.lan.
  3. Enter root password of your Mac so /private/etc/hosts can be updated.

Vagrant, Shell Provisioner, shell script, and Ansible

Next, we will fire up 3rd Vagrant VM, web3.vag.lan, using Shell Provisioner (a feature within Vagrant) and setup1.sh. When you use Shell Provisioner and setup01.sh, the Vagrant VM will be ready with Ansible client as soon as it boots up. No manual configuration is required to install Ansible client.

Make sure you are in Terminal01. Check you are at macOS shell prompt.

Create a directory for web3.vag.lan0. Copy Vagrantfile from web2/ into web3/.

mkdir ~/codes/vagrants/web3
cp ~/codes/vagrants/web2/Vagrantfile ~/codes/vagrants/web3/

We are going to insert following 2 lines into ~/codes/vagrants/web3/Vagrantfile, below line 12.


  config.vm.provision "shell",
    path: "~/codes/scripts/setup1.sh"

Note the path to setup1.sh is absolute path.

You will need to update the IP and change the hostname to web3.vag.lan.

After editing, ~/codes/vagrants/web3/Vagrantfile should look like below.

In all you added 2 new lines. And modified 3 existing lines.

# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # General Vagrant VM configuration.
  config.vm.box = "centos/7"
  config.ssh.insert_key = true
  config.vm.synced_folder ".", "/vagrant", disabled: true
  config.vm.provision "shell",
    path: "~/codes/scripts/setup1.sh"
  config.vm.provider :virtualbox do |v|
    v.memory = 512
    v.linked_clone = true
  end

  # Server 1.
  config.vm.define "web3.vag.lan" do |web|
    web.vm.hostname = "web3.vag.lan"
    web.vm.network :public_network, ip: "192.168.11.213", bridge: "en0: Wi-Fi (Airport)", bootproto: "static", gateway: "192.168.11.1"
  end
end

    Make sure you are in directory ~/codes/vagrants/web3/.

    Use vagrant up to start web3.vag.lan.

    Enter root password of your Mac when prompted.

    You will see a lot more text scroll by, versus when starting web1 and web2. These are for tasks done by setup1.sh.

    When web3.vag.lan boots up the 1st time, it already has Ansible client installed.

Test Ansible with web3.vag.lan

Switch to Terminal03, which has Ansible virtualenv activated.

Let's test managing web3.vag.lan with the Ansible controller on your Mac.

With this VM, we will not use ~/codes/ansible/hosts for the list of Ansible clients. Instead we will create a new folder and keep the Ansible hosts file in the new folder. In this tutorial it would be "~/codes/ansible/hosts-file/provisioner.vag".

As I started using Ansible, I found that I didn't always update ~/codes/ansible/hosts before starting to type ansible command. I should, but I didn't. Having separate a hosts file for different groups of Ansible clients made it easier to avoid making wrong changes to wrong servers.

So let's create the folder to keep Ansible hosts file.

mkdir ~/codes/ansible/hosts-file

Next, create file ~/codes/ansible/hosts-file/provisioner.vag with following. Again, only difference is the last line.

[all:vars]
remote_user = ansible
ansible_ssh_private_key_file = /Users/paul/.ssh/ansible_mac

[webservers]
web3.vag.lan

Now we are ready to test Ansible with web3.vag.lan.

  1. Check you are at Terminal03, where Ansible virtualenv is already activated.
  2. In Terminal03, run Ansible command.
  3. ansible all -m ping -i ~/codes/ansible/hosts-file/provisioner.vag
    
  4. The only difference here is the last part -i ~/codes/ansible/hosts-file/provisioner.vag
  5. The -i key tells Ansible to use a specified file as the new hosts file. Effectively -i key value replaces ~/codes/ansible/hosts with a new hosts file.
  6. If everything is working, you should see following.
  7. (ansible)MacBook:~ paul$ ansible all -m ping -i ~/codes/ansible/hosts-file/provisioner.vag
    web3.vag.lan | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    
  8. And there it is. Using Vagrant, Shell Provisioner plugin of Vagrant, and a shell script, we got a CentOS 7 instance with Ansible ready to go in a minute, with no manual configuring.

By modifying a Vagrantfile, we could fire up 10 or 100 VMs (if enough RAM is available) in a few minutes.

Let's delete web3.vag.lan.

  1. Switch to Terminal01.
  2. Log out from any VM.
  3. In Mac's shell prompt, change directory to ~/codes/vagrants/web3/
  4. Run "vagrant destroy -f" to delete VM web3.vag.lan.
  5. Enter root password of your Mac so /private/etc/hosts can be updated.

Test Ansible with 3 web servers simultaneously

  1. Check you are in Terminal01. Make sure you are logged out of any VM, and at macOS prompt.
  2. Lets use Vagrant and bring up 3 web servers simultaneously.
  3. Prepare Vagrantfile
  4. mkdir ~/codes/vagrants/web4-6
    cp ~/codes/vagrants/web3/Vagrantfile ~/codes/vagrants/web4-6/
    
  5. Edit ~/codes/vagrants/web4-6/Vagrantfile so it looks like below. Basically we duplicate "Server 1" twice. And update all 3 servers with new hostname and IP.
  6. # -*- mode: ruby -*-
    # vi: set ft=ruby :
    VAGRANTFILE_API_VERSION = "2"
    # All Vagrant configuration is done below. The "2" in Vagrant.configure
    # configures the configuration version (we support older styles for
    # backwards compatibility). Please don't change it unless you know what
    # you're doing.
    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
      # General Vagrant VM configuration.
      config.vm.box = "centos/7"
      config.ssh.insert_key = true
      config.vm.synced_folder ".", "/vagrant", disabled: true
      config.vm.provision "shell",
        path: "~/codes/scripts/setup1.sh"
      config.vm.provider :virtualbox do |v|
        v.memory = 512
        v.linked_clone = true
      end
    
      # Server 1.
      config.vm.define "web4.vag.lan" do |web|
        web.vm.hostname = "web4.vag.lan"
        web.vm.network :public_network, ip: "192.168.11.214", bridge: "en0: Wi-Fi (Airport)", bootproto: "static", gateway: "192.168.11.1"
      end
    
      # Server 2.
      config.vm.define "web5.vag.lan" do |web|
        web.vm.hostname = "web5.vag.lan"
        web.vm.network :public_network, ip: "192.168.11.215", bridge: "en0: Wi-Fi (Airport)", bootproto: "static", gateway: "192.168.11.1"
      end
    
      # Server 3.
      config.vm.define "web6.vag.lan" do |web|
        web.vm.hostname = "web6.vag.lan"
        web.vm.network :public_network, ip: "192.168.11.216", bridge: "en0: Wi-Fi (Airport)", bootproto: "static", gateway: "192.168.11.1"
      end
    
    end
    
  7. In Terminal01, check you are in directory ~/codes/vagrants/web4-6/.
  8. Run vagrant up. It will take longer as Vagrant is bring up 3 VMS while running shell script in each VM.
  9. Check status of the VMs with vagrant status and you should see something like below.
  10. 
    MacBook:web4-6 paul$ vagrant status
    Current machine states:
    
    web4.vag.lan              running (virtualbox)
    web5.vag.lan              running (virtualbox)
    web6.vag.lan              running (virtualbox)
    ...
    
  11. Note all 3 VMs are ready with Ansible client.
  12. Next, we need to prepare Vagrant hosts file.
  13. Switch to Terminal03.
  14. Copy file ~/codes/ansible/hosts-file/provisioner.vag to ~/codes/ansible/hosts-file/web4-6.vag
  15. Update Ansible hosts file, ~/codes/ansible/hosts-file/web4-6.vag, so that it looks like below.
  16. [all:vars]
    remote_user = ansible
    ansible_ssh_private_key_file = /Users/paul/.ssh/ansible_mac
    
    [webservers]
    web4.vag.lan
    web5.vag.lan
    web6.vag.lan
    
  17. In Terminal03, let's run ansible command and execute "hostname" on all 3 servers.
  18. ansible all -a "hostname" -i ~/codes/ansible/hosts-file/web4-6.vag
    
  19. You would see something like below.
  20. (ansible)MacBook:web4-6 paul$ ansible all -a "hostname" -i ~/codes/ansible/hosts-file/web4-6.vag
    web6.vag.lan | SUCCESS | rc=0 >>
    web6.vag.lan
    
    web5.vag.lan | SUCCESS | rc=0 >>
    web5.vag.lan
    
    web4.vag.lan | SUCCESS | rc=0 >>
    web4.vag.lan
    
  21. Note the names of the servers in the output are not sorted. If you run it a few times, you will notice the order changes. This is because Ansible controller by default starts 5 separate processes to execute the command. This is for faster response. If you'd rather get the result sorted consistently, use "-f1" option. That will start only 1 process and the outputs will be sorted consistently, as shown below.
  22. (ansible)MacBook:web4-6 paul$ ansible all -a "hostname" -f1 -i ~/codes/ansible/hosts-file/web4-6.vag
    web4.vag.lan | SUCCESS | rc=0 >>
    web4.vag.lan
    
    web5.vag.lan | SUCCESS | rc=0 >>
    web5.vag.lan
    
    web6.vag.lan | SUCCESS | rc=0 >>
    web6.vag.lan
    

Let's delete the 3 VMS.

  1. Switch to Terminal01.
  2. Log out from any VM.
  3. In Mac's shell prompt, change directory to ~/codes/vagrants/web4-6/
  4. Run "vagrant destroy -f" to delete all 3 VMs.
  5. Enter root password of your Mac so /private/etc/hosts can be updated.

That's about it. In next blog, I will show how to use setup1.sh with an Amazon AWS EC2 instance to allow provisioning multiple EC2 instances with Ansible client enabled.