Using KVM, Kickstart, SaltStack to provision multiple CentOS rapidly

Posted by paul on 2015.05.08

Using KVM/Kickstart/SaltStack to provision multiple CentOS rapidly

Created 2014.06.06 Fri

Updated 2015.05.08 Tue

DigitalOcean and Linode and other VPS hosting companies are popular because they let you provision new Linux hosts quickly, like one in a minute. Using open source tools, you can have exact same ability to set up new Linux hosts quickly in your own LAN, albeit without the fancy web interface. And all free.

Every Linux sysadmin (and even non-sysadmins like coders and DBAs) need to set up a new Linux host once in a while. The problem is it is a 10-30min long process (depending on the hardware). You start installation, answer some questions, wait until install is done, and come back to finish up. Even if your hardware is a monster and can handle file copy/installation in a minute, you still need to sit and answer some prompts. If you needed to set up 10 new hosts, it gets bad. This is why DigitalOcean and Linode are popular. They let you provision a new Linux host in a minute or so. While such VPS services are great, you sometimes just want to set up new Linux hosts in your own environment.

On this web page, I will describe how you can provision new Linux hosts very quickly and easily using automation. And with SaltStack (hereafter Salt) in place, you don't ever have to log into the newly installed Linux host with ssh. You can administer it remotely using Salt. And this can be replicated pretty much endlessly, as long as your hardware can keep up. Setting up a new dev/qa environment (even 50 hosts) will no longer be a project, but one bullet point on your daily todo list. It will be like having DigitalOcean allowing you to use provisioning tools in your private environment, minus the fancy web GUI.

I first wrote first version of this tutorial in mid 2014. The 2014 version is still available. The updated 2015 version here includes some updates, including using kickstart-prep.sh to generate .sh and .ks files, needed by Kickstart. And also Salt, which helps ease administering many Linux hosts immensely.

Overview

Some assumptions and tools needed.

  1. OS: All computers run CentOS 6.x x86_64. This tutorial was written using 6.6.
  2. KVM Host: Required. r2d2.base.loc is the KVM host, physical box running CentOS 6.6 x86_64. Its IP in this tutorial is 192.168.12.10.
  3. IP setting: My Wireless Router's DHCP server gives out following: IP 192.168.12.xxx Netmask 255.255.255.0 Gateway 192.168.12.1. DNS server is 192.168.12.9.
  4. Local Yumrepo: Required. You need a working local Yumrepo server. In this tutorial, it's assumed to be running on 192.168.12.111.
  5. Local DNS server: Optional. If you have no local DNS server in your home/office LAN yet, I highly recommend that you set it up before starting on this tutorial. You can use my tutorial posted last year. Using Salt without a local DNS server is possible, unlike Puppet. However having a local DNS server makes setting up Salt a bit earlier. If you find setting up a Local DNS service too hard to complete, you can always skip it, complete this tutorial, and return to setting up a local DNS service later.
  6. Salt: Optional. Setting up Salt is really simple. You can follow my tutorial. While having Salt is optional, it will help you immensely with automated administration of your Linux environment. Because setting it up is so easy, I suggest you set up Salt before proeeding.
  7. IPs: Following KVM Guest being installed.
    1. lab-web-tmpl-01: 192.168.12.150. Installed so you can use the file, anaconda-ks.cfg.
    2. labweb01: 192.168.12.151. First host to be installed with Kickstart.
    3. labweb02: 192.168.12.152. Second host to be installed, with modified Kickstart.
    4. labweb03: 192.168.12.153. Third host to be installed, with Kickstart config files generated automatically from kickstart-prep.sh that I coded.

Prerequisites to complete

  1. Before starting, you first need to make sure you have a working KVM Host. I posted the tutorial to set it up. If you are trying to learn about Linux, setting up a KVM host should be first on your to do list, as soon as you become familiar with basic command line and vi editor. It will provide much greater flexibility.
  2. Next you need a working local Yum Repository server, and I also posted the tutorial for it. Most of the time needed to finish setting up a Local Yum Repository server is for waiting for 10GB of files to be downloaded from internet. In my tutorial, I included a script that automates most of the steps.
  3. Local DNS service is recommended but not required to complete this turorial. If you would like to set up a local DNS service so that you can reach your server by going to labweb01 instead of typing out the full IP address, you should set it up. I posted the tutorial also.

Now you are ready. It's true that setting up KVM host, local Yum Repository server, and DNS service (option) will take a good few days to complete (only an hour or so of actual work and rest is waiting for files to download). But the investment you make here will pay off in saved time and less hassle.

Install a template OS, lab-web-tmpl-01

You will first start out by setting up a template KVM Guest, mainly to use a file (anaconda-ks.cfg) generated after a CentOS is installed. Let's first install a KVM Guest OS to use as a template OS. Root password, drive partition, and packages you select for the template OS will be used on other KVM Guests that will be installed with Kickstar later. They can be easily modified later, but to generate from scratch, it's best to install a template host.

  1. Mac: On your Mac, open Terminal and ssh log into r2d2. It's important to use -Y option. Use IP address if you have no local DNS service.
  2. ssh -Y [email protected]
    
  3. Mac: In the Terminal while logged into r2d2, run command virt-manager as below and a virt-manager GUI will open on your Mac's screen.
  4. [[email protected] ~]# virt-manager
    
  5. virt-manager: Click on "New" to start installation process. Name the new host as lab-web-tmpl-01. I recommend keeping the hostname and the Name of the KVM Guest to be the same for easier administration. This new Linux host will be a KVM Guest running on the KVM Host, r2d2.
  6. virt-manager: Pick Network install (HTTP, FTP, or NFS). This way you will be able to test the local Yum Repo server also. You could use local CDROM, but having a local Yum Repo server will open up a lot of possibilities in automating your Linux environment.
  7. virt-manager: Step 2 of 5. For URL, in my environment I enter http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/. If you followed my tutorial for setting up your local Yum Repository server for CentOS 6.6, you would use the same URL, except for the IP. Note that I am using IP address, not FQDN. I've had some issues with using FQDN when using Kickstart, so I use IP here to specify the local Yum Repo server. In all other places, using FQDN is fine.
  8. Because virt-manager GUI running on Mac is X11 window, you cannot use Cmd+v to paste text in. You can however right click and choose Paste command.
  9. Remember that the root password you enter in this install will be used in all following KVM Guests installed using this Kickstart file. If you use Salt, you very rarely will have to use the root password as Salt will not require typing in root password on the KVM Guests. In some rare cases you will need it though, so write it down.
  10. virt-manager: Click on Forward.
  11. virt-manager: Step 3 of 5. Select RAM and CPU. Note that these 2 are not configured in Kickstart file, but rather when you issue virt-install command, which you will see later. Click on Forward.
  12. virt-manager: Step 4 of 5. Change the disk size from default 8.0G to what you prefer. Uncheck Allocate entire disk now which will speed up the install process.
  13. virt-manager: Step 4 of 5. You should also specify where the file containing the virtual hd of lab-web-tmpl-01 will be stored. By default it will be in /var/lib/libvirt/images/ of the KVM Host. However I prefer keeping the virtual disk files in /vm/ of the KVM Host, as explained in my tutorial on setting up KVM Host. To do so, click on Select managed or other existing storage and type in /vm/lab-web-tmpl-01.img. Click on Forward.
  14. virt-manager: Step 5 of 5. Check Customize configuration before install and click on Finish.
  15. virt-manager: Click on Display VNC . Next pick correct value for Keymap. I am using English US keyboard, so I pick en-us. Click on Apply.
  16. virt-manager: Click on Begin Installation and the installation of the new KVM Guest, lab-web-tmpl-01 will begin.
  17. lab-web-tmpl-01: Important Click on View > Scale Display > Always. Without this done, some screen will be cut off and you won't be able to continue installing the new KVM Guest.
  18. lab-web-tmpl-01: Next you will need to provide some answers to the installer of CentOS. Pick your language, timezone, etc. Use tab, spacebar, and Enter key to navigate until you can use your mouse.
  19. lab-web-tmpl-01: Pick Basic Storage Devices, discard any data when asked about the Virtio Block device, accept default hostname, accept time zone, enter root password (make sure to remember this. using kickpassword for this example), use all disk space. The virtual disk will be partitioned and formatted.
  20. lab-web-tmpl-01: To pick software to install, pick Minimal or Basic Server . I will go with Minimal for faster install.
  21. lab-web-tmpl-01: The actual install of lab-web-tmpl-01 will begin.
  22. lab-web-tmpl-01: Reboot lab-web-tmpl-01 after the installation is done.

Examine anaconda-ks.cfg

You need to copy anaconda-ks.cfg from lab-web-tmpl-01 to r2d2, and modify it for use as a Kickstart file. But first you need to create a folder and copy some files on r2d2.

  1. lab-web-tmpl-01: In virt-viewer, log in to lab-web-tmpl-01 and determine its current IP with ifconfig command. I will assume its IP is 192.168.12.237, which was from the DHCP server on my home wireless router.
  2. lab-web-tmpl-01: Install openssh-client RPM, in order to install scp. I installed using Minimal so some additional RPMs need to be installed.
  3. [[email protected] ~]# yum install -y openssh-clients
    
  4. r2d2 (KVM host): Run scp command on r2d2 to copy lab-web-tmpl-01:/root/anaconda-ks.cfg to r2d2:/root/. Rename the file to something ending with .ks extension while copying.
  5. # Change source IP to what matches your environment.
    [[email protected] ~]# scp [email protected]:/root/anaconda-ks.cfg ~/lab-web-tmpl-01.ks
    
  6. r2d2: /root/lab-web-tmpl-01.ks should look like below. The code in plain text.
  7. # Kickstart file automatically generated by anaconda.
    
    #version=DEVEL
    install
    url --url=http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/
    lang en_US.UTF-8
    keyboard us
    network --onboot yes --device eth0 --bootproto dhcp --noipv6
    rootpw  --iscrypted $6$Sav9fJOd7WXcj8Qd$91pvA5fawHfSzJhdXf97L9lnCXuIdpXbvH4NZR1NkMEgkFqftBb4k9mslJjOP17dyvu9FYdhcH9m1y3xYJbbi0
    firewall --service=ssh
    authconfig --enableshadow --passalgo=sha512
    selinux --enforcing
    timezone --utc Etc/UTC
    bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
    # The following is the partition information you requested
    # Note that any partitions you deleted are not expressed
    # here so unless you clear all partitions first, this is
    # not guaranteed to work
    #clearpart --all --drives=vda
    #volgroup VolGroup --pesize=4096 pv.253002
    #logvol / --fstype=ext4 --name=lv_root --vgname=VolGroup --grow --size=1024 --maxsize=51200
    #logvol swap --name=lv_swap --vgname=VolGroup --grow --size=921 --maxsize=921
    
    #part /boot --fstype=ext4 --size=500
    #part pv.253002 --grow --size=1
    
    repo --name="CentOS"  --baseurl=http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/ --cost=100
    
    %packages
    @core
    @server-policy
    @workstation-policy
    %end
    

Edit labweb01.ks file to install labweb01

Before beginning to edit the .ks file to prepare to install labweb01, create a folder and copy some files.

  1. r2d2: Create directory, /root/ks/
  2. r2d2: Copy /root/lab-web-tmpl-01.ks > /root/ks/labweb01.ks.
  3. [[email protected] ~]# mkdir /root/ks
    [[email protected] ~]# mv /root/lab-web-tmpl-01.ks /root/ks/
    [[email protected] ~]# cd /root/ks
    [[email protected] ~]# cp lab-web-tmpl-01.ks labweb01.ks
    

On r2d2, edit /root/ks/labweb01.ks.

  1. Add: Add text under install. It configures CentOS installer to run in text mode.
  2. #OLD
    install
    
    #NEW
    install
    text
    
  3. Replace: Replace the line starting with network with following. Change the IP settings to match your setting. The important change is IP will be static, not assigned by DHCP server.
  4. #OLD
    network --onboot yes --device eth0 --bootproto dhcp --noipv6
    
    #NEW
    network --onboot yes --device eth0 --bootproto static --ip=192.168.12.151 --netmask 255.255.255.0 --gateway 192.168.12.1 --nameserver=192.168.12.11 --ipv6 auto
    
  5. Update: Update 2 lines. Firewall will be enabled later using Salt.
  6. #OLD
    firewall --service=ssh
    selinux --enforcing
    
    #NEW
    firewall --service=disabled
    selinux --disabled
    
  7. Add: Add following 4 lines below bootloader --location-bmr.... It takes care of all partitioning options.
  8. #OLD
    bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
    
    #NEW
    bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
    
    zerombr
    # Partition table
    clearpart --all
    autopart
    
  9. Add: Add 1 line to have labweb01 reboot automatically after the installation is done. The line needs to be inserted right below %end.
  10. #OLD
    %end
    
    #NEW
    %end
    
    reboot
    
  11. r2d2: At this point, the file /root/ks/labweb01.ks should look like below. Remember to update url and IP address to match your environment. The code labweb01.ks in plain text. Do NOT copy/paste the code below as some text will get cut off and not work.
  12. # Kickstart file automatically generated by anaconda.
    
    #version=DEVEL
    install
    text
    url --url=http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/
    lang en_US.UTF-8
    keyboard us
    network --onboot yes --device eth0 --bootproto static --ip=192.168.12.151 --netmask 255.255.255.0 --gateway 192.168.12.1 --nameserver=192.168.12.11 --ipv6 auto
    rootpw  --iscrypted $6$Sav9fJOd7WXcj8Qd$91pvA5fawHfSzJhdXf97L9lnCXuIdpXbvH4NZR1NkMEgkFqftBb4k9mslJjOP17dyvu9FYdhcH9m1y3xYJbbi0
    firewall --service=disabled
    authconfig --enableshadow --passalgo=sha512
    selinux --disabled
    timezone --utc Etc/UTC
    bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
    
    zerombr
    # Partition table
    clearpart --all
    autopart
    
    # The following is the partition information you requested
    # Note that any partitions you deleted are not expressed
    # here so unless you clear all partitions first, this is
    # not guaranteed to work
    #clearpart --all --drives=vda
    #volgroup VolGroup --pesize=4096 pv.253002
    #logvol / --fstype=ext4 --name=lv_root --vgname=VolGroup --grow --size=1024 --maxsize=51200
    #logvol swap --name=lv_swap --vgname=VolGroup --grow --size=921 --maxsize=921
    
    #part /boot --fstype=ext4 --size=500
    #part pv.253002 --grow --size=1
    
    repo --name="CentOS"  --baseurl=http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/ --cost=100
    
    %packages
    @core
    @server-policy
    @workstation-policy
    
    %end
    reboot
    
  13. With the .ks file ready, you are now ready to create a shell script labweb01.sh that call virt-install. Virt-install command is what creates a new KVM Guest on KVM Guest.

Create labweb01.sh to install labweb01

When you were installing lab-web-tmpl-01 in virt-manager GUI tool earlier, you were clicking around to do it. There is an equivalent tool in command line, virt-install. Virt-install creates a virtual machine within the KVM Host and begins the installation of the KVM Guest on top of the virtual machine. Virt-install command requires about 13 options, meaning you should probably create a script to use virt-install.

  1. r2d2: Create file /root/ks/labweb01.sh with following. You can save the file anywhere but I keep it with the Kickstart files. Verify the value of variable Yumrepo matches your environment. The code in plain text.
  2. #!/bin/bash
    
    Host1="labweb01"
    Yumrepo="http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64"
    
    virt-install \
    -w bridge=br0 \
    --connect qemu:///system \
    -n $Host1 \
    --disk /vm/$Host1.img,size=8 \
    -r 1024 \
    --vcpus=1 \
    --arch=x86_64 \
    -l $Yumrepo \
    --os-type=linux \
    --os-variant=rhel6 \
    --noautoconsole \
    --wait=-1 \
    --keymap=en-us \
    --initrd-inject=/root/ks/$Host1.ks \
    -x "ks=file:/$Host1.ks"
    
    1. -w bridge=br0 --> This sets the KVM Guest to use the bridge network of KVM Host, ensuring the KVM Guest has full inbound/outbound network access.
    2. -n labweb01 --> Names the KVM Guest as labweb01 within the KVM environment. You could use any name but it's best to use same name for both the hostname and the KVM Guest name.
    3. --disk /vm/labweb01.img,size=8 --> creates virtual disk image at /vm/labweb01.img and sets the size to 8GB.
    4. -r 1024 --> Sets the memory to 1024MB.
    5. --vcpus=1 --> sets to use 1 virtual CPU.
    6. --arch=x86_64 --> sets to use 64bit architecture.
    7. -l $Yumrepo --> Source of install RPMs (ie. rpms on CentOS install DVDs). This is on the local Yumrepo server. Change it to match your environment.
    8. --keymap=en-us --> sets the keyboard to US English mapping.
    9. --noautoconsole --> prevents virt-viewer window from opening automatically when an installation starts. We wouldn't want to have to close 30 virt-viewer windows when installing 30 KVM Guests.
    10. --wait=-1 --> necessary for automatically rebooting newly installed KVM Guest after installation is done.
    11. --initrd-inject=/root/ks/labweb01.ks --> tells virt-install command where to look for labweb01.ks, the Kickstart file. It's the actual file path of labweb01.ks.
    12. -x "ks=file:/labweb01.ks --> tells virt-install command to use labweb01.ks as the Kickstart file. Note the / at beginning of /labweb01.ks. That's not a typo. And no, you do not use the full file path here.
  3. Save /root/ks/labweb01.sh and make it executable.
  4. [[email protected] ~]# chmod 750 /root/ks/labweb01.sh
    

Install labweb01

Now you are ready to install labweb01 using Kickstart, which consists of executing script labweb01.sh, which uses config values in file labweb01.ks.

  1. Your Mac: If not logged into r2d2 from your Mac yet, ssh log into r2d2 as root, using -Y option. You must use -Y option.
  2. ssh -Y [email protected]
    
  3. r2d2: Execute /root/ks/labweb01.sh &. Using & option puts the process in the background and allows you to issue other commands. You can always open a new Terminal window and log into r2d2.
  4. r2d2: You should see following in Terminal.
  5. [[email protected] ~]# /root/ks/labweb01.sh &
    [1] 27995
    [[email protected] ~]#
    Starting install...
    Retrieving file vmlinuz...                                                          | 7.9 MB     00:00 ...
    Retrieving file initrd.img...                                                       |  66 MB     00:00 ...
    Creating storage file labweb01.img                                                  |  19 GB     00:00
    Creating domain...                                                                  |    0 B     00:00
    Domain installation still in progress. Waiting for installation to complete.
    
    
  6. virt-install command needs to hang around until the installation is finished and then virt-install command will initiate reboot of the newly installed KVM Guest. Hence using & option to put the process in background.
  7. Your Mac: switch to virt-manager and double click on labweb01 to view the installation.
  8. You should see a new window (probably named labweb01 (1) - Virt Viewer). This is the virtual screen of labweb01. You will note you do not need to provide any answers to prompts as all installation options are answered by 2 files: labweb01.ks AND labweb01.sh.
  9. Wait about 1-30 min (depending on your hardware), and labweb01 will reboot and be ready with a static IP address, in this case 192.168.12.151.
  10. With KVM/Kickstart, provisioning a new Linux is fully automated. You did not have to set down and answer to prompts for 3 min, wait for installation to finish and return to configure IP. All of it was done for you.
  11. Next, I will show how to modify labweb02.ks and labweb02.sh to install labweb02. I will also integrate some additional configs to install Salt Minion on labweb02 using Kickstart.

Prepare to install labweb02 and add Salt Minion

Here, I will set up a second KVM Guest, labweb02, and in the process show how to configure the .ks file so that Salt Minion is automatically installed as labweb02 is installed with Kickstart. Salt will allow you to remotely administer the newly installed KVM Guests as soon as the installation is done. You won't ever have to log into each newly installed KVM Guest. This is crucial when you are installing many Linux hosts.

  1. r2d2: Copy /root/ks/labweb01.sh to /root/ks/labweb02.sh.
  2. r2d2: Copy /root/ks/labweb01.ks to /root/ks/labweb02.ks.
  3. r2d2: Open /root/ks/labweb02.sh and change value of $Host1 to labweb02.
  4. # BEFORE
    Host1="labweb01"
    
    # AFTER
    Host1="labweb02"
    
  5. r2d2: Edit /root/ks/labweb02.ks and change following value. Basically labweb01 and labweb02 have same configs, except for IP address as expected. Note root password on both labweb01 and labweb02 remain the same, which was set to kickpassword during the installation of lab-web-tmpl-01 earlier.
    1. Change --ip=192.168.12.151 to --ip=192.168.12.152
  6. r2d2: Edit /root/ks/labweb02.ks and add following lines before %end.
  7. %post
    # 1 Optional. Update /etc/resolv.conf
    sed -i '1s/^/search base.loc\n/' /etc/resolv.conf
    
    # 2 Update /etc/sysconfig/network
    sed -i "/HOSTNAME=/c\HOSTNAME=lab02.base.loc" /etc/sysconfig/network
    
    # 3 Install salt-minion and change Salt ID
    rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum -y install salt-minion
    sed -i "s/#id:/id: lab02.base.loc/" /etc/salt/minion
    
    # 4 Update Master Server name by commenting out 4a or 4b.
    # Change saltserver.base.loc to match yours. You can also replace it with the IP address of your Salt Master.
    # If your Salt Master is reachable with 'salt' because you have DNS service running in your LAN,
    # you can comment out both 4a and 4b.
    sed -i "/#master:/c\master: saltserver.base.loc" /etc/salt/minion
    
    #5 Start salt-minion service
    chkconfig salt-minion on
    service salt-minion restart
    
    1. #1. Optional. /etc/resolv.conf --> is updated so lab01 can reach Salt Master by just looking for 'salt'. Change "base.loc" to match your environment.
    2. #2. /etc/sysconfig/network --> is updated so that it's easy to identify different CentOS computers when logging in.
    3. #3. EPEL repo is added and Salt-Minion is installed. Also Salt ID is specified to match the hostname.
    4. #4. saltserver.base.loc --> Specify hostname of the Salt Master to look for. Change saltserver.base.loc to match yours. You can use IP. If your Salt Master is reachable via 'salt', you can skip this. Salt Minion looks for 'salt' by default.
  8. Here's is what /root/ks/labweb02.ks should look like at this point. The code of labweb02.ks in plain text is here. Please do NOT copy/paste the code below. Some text will get separated and will not work.
  9. # Kickstart file automatically generated by anaconda.
    
    #version=DEVEL
    install
    text
    url --url=http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/
    lang en_US.UTF-8
    keyboard us
    network --onboot yes --device eth0 --bootproto static --ip=192.168.12.152 --netmask 255.255.255.0 --gateway 192.168.12.1 --nameserver=192.168.12.11 --ipv6 auto
    rootpw  --iscrypted $6$Sav9fJOd7WXcj8Qd$91pvA5fawHfSzJhdXf97L9lnCXuIdpXbvH4NZR1NkMEgkFqftBb4k9mslJjOP17dyvu
    9FYdhcH9m1y3xYJbbi0
    firewall --service=disabled
    authconfig --enableshadow --passalgo=sha512
    selinux --disabled
    timezone --utc Etc/UTC
    bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
    
    zerombr
    # Partition table
    clearpart --all
    autopart
    
    
    # The following is the partition information you requested
    # Note that any partitions you deleted are not expressed
    # here so unless you clear all partitions first, this is
    # not guaranteed to work
    #clearpart --all --drives=vda
    #volgroup VolGroup --pesize=4096 pv.253002
    #logvol / --fstype=ext4 --name=lv_root --vgname=VolGroup --grow --size=1024 --maxsize=51200
    #logvol swap --name=lv_swap --vgname=VolGroup --grow --size=921 --maxsize=921
    
    #part /boot --fstype=ext4 --size=500
    #part pv.253002 --grow --size=1
    
    repo --name="CentOS"  --baseurl=http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/ --cost=100
    
    %packages
    @core
    @server-policy
    @workstation-policy
    
    %post
    # 1 Optional. Update /etc/resolv.conf
    sed -i '1s/^/search base.loc\n/' /etc/resolv.conf
    
    # 2 Update /etc/sysconfig/network
    sed -i "/HOSTNAME=/c\HOSTNAME=lab02.base.loc" /etc/sysconfig/network
    
    # 3 Install salt-minion and change Salt ID
    rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum -y install salt-minion
    sed -i "s/#id:/id: lab02.base.loc/" /etc/salt/minion
    
    # 4 Update Master Server name by commenting out 4a or 4b.
    # Change saltserver.base.loc to match yours. You can also replace it with the IP address of your Salt Maste
    r.
    # If your Salt Master is reachable with 'salt' because you have DNS service running in your LAN,
    # you can comment out both 4a and 4b.
    sed -i "/#master:/c\master: saltserver.base.loc" /etc/salt/minion
    
    #5 Start salt-minion service
    chkconfig salt-minion on
    service salt-minion restart
    
    %end
    
    reboot
    
  10. Lines between %post and %end are executed AFTER the basic Linux OS is installed.

Install labweb02 and test Salt Minion works

Verify following values before installing labweb02.

  1. labweb02.ks: Check --ip value is correct for labweb02
  2. labweb02.ks: Check lab02.home.loc is used in %post section
  3. labweb02.ks: Verify value saltserver.base.loc is actually the hostname or IP of your Salt Master.
  4. labweb02.sh: Check value of Host1 is "labweb02"

Let's install labweb02 by running /root/ks/labweb02.sh. Most importantly, Salt Minion will be installed and configured on labweb02. Immediately after labweb02 is rebooted, you should be able to remotely administer labweb02 from your Salt Master. Everything that is required to set up Salt Minion on labweb02 is already done by labweb02.ks. This will save huge amount of time/energy when you are provisioning many KVM Guests later.

Salt Master: Once labweb02 is installed, on your Salt Master server look for the public key sent by Salt Minion on labweb02. Initially it will be an unaccepted key, as shown below. Accept the key with the salt-key command as shown below.

[[email protected] ~]# salt-key -L
Accepted Keys:
labfiler1.home.loc
...
Unaccepted Keys:
labweb02.base.loc
Rejected Keys:
[[email protected] ~]#
[[email protected] ~]# salt-key -a labweb02*
Unaccepted Keys:
labweb02.home.loc
Proceed? [n/Y] y
Key for minion labweb02.base.loc accepted.
[[email protected] ~]#

Next, test issuing commands from your Salt Master to labweb02. You should see following.

[[email protected] ~]#  salt 'labweb02*' test.ping
labweb02.base.loc:
    True

[[email protected] ~]#  salt 'labweb02*' cmd.run 'date'
labweb02.base.loc:
    Wed May  6 22:48:08 UTC 2015

At this point, you are able to provision a KVM Guest simply by running a .sh file and walk away. The .sh and .ks files take care of everything. It even adds Salt Minion, greatly easing administering the KVM Guests. One trouble you may have noticed is you had to edit 2 files (labweb02.ks and labweb02.sh) manually to prepare to install labweb02. Not very scalable and prone to errors. To automate creating the 2 files, we will create a shell script, kickstart-prep.sh, that will generate a .ks and a .sh for each KVM Guest.

Kickstart-prep.sh

  1. r2d2: To automate creating .ks and .sh files, you will create a script, /root/bin/kickstart-prep.sh. Copy code from kickstart-prep.sh. Do not copy/paste the code below as some text will break up and not work.
  2. It's rather long. When you run kickstart-prep.sh, you will be prompted for some values. Let's say you are naming the Linux host as labweb03 and provided answers to other questions, it will generate 2 files (labweb03.sh and labweb03.ks). You then execute labweb03.sh and a KVM Guest named labweb03 will be installed, complete with Salt Minion.
  3. #!/bin/bash
    
    # This script will generate .ks and .sh files used in Kickstart installations.
    
    #####
    # prompt for variables that should be updated
    #####
    
    echo "Pick CentOS version."
    echo "1: CentOS 6.6"
    echo "2: CentOS 6.5"
    
    
    read -e -p "Please enter your choice:  " Choice
    
    if [ $Choice == 1 ]; then
    VERSION1="6.6"  #Ex.: 6.5  OR 6.6
    elif [ $Choice == 2 ]; then
    VERSION1="6.5"  #Ex.: 6.5  OR 6.6
    else
    echo "Wrong option picked. Existing the script."
    exit 1
    fi
    
    
    #####
    # Variables
    #####
    HOST1="server01"  # Name of .sh and .ks files to be created.
    TargetDir="/root/ks"  # Directory where .sh and .ks files will be created in.
    DOMAIN="home.loc"  # domain the HOST1 is in.
    YUMREPO="http://192.168.12.113/yumrpms/centos$VERSION1/$VERSION1/os/x86_64/"  # Yumrepo server with rpms
    IPADDR="192.168.12.151"
    GATEWAY="192.168.12.1"
    NETMASK="255.255.255.0"
    NAMESERVER1="192.168.12.11"
    SaltMaster="192.168.12.12"
    DiskSize="8"
    
    
    #####
    # prompt for values
    #####
    echo "########################################################"
    echo "Provide values your server will use. Just hit Enter to accept default value."
    echo "Hit ctrl-c to escape the script if necessary."
    echo " "
    read -e -p "Hostname (short hostname as server01): " -i "$HOST1" HOST1
    echo " "
    echo " "
    read -e -p "Virtual Disk size to use: " -i "$DiskSize" DiskSize
    echo " "
    echo "Domain the $HOST1 is in: "
    read -e -p "If the FQDN is $HOST1.home.loc, enter home.loc:   " -i "home.loc" DOMAIN
    echo " "
    read -e -p "Static IP:   " -i "$IPADDR" IP
    echo " "
    read -e -p "Gateway:   " -i "$GATEWAY" GATEWAY
    echo " "
    read -e -p "Netmask:   " -i "$NETMASK" NETMASK
    echo " "
    read -e -p "IP of the nameserver:   " -i "$NAMESERVER1" NAMESERVER1
    echo " "
    read -e -p "IP or hostname of the Salt Master:   " -i "$SaltMaster" SaltMaster
    echo " "
    
    
    
    #####
    # Create %HOST1.ks file
    #####
    
    # The $ character in the value of rootpw below is escaped with '\'. Escaping $ with '\' is required when using <<EOM.  In actual .ks file '\$' will be converted to '$'.
    
    echo " "
    cat <<EOM > $TargetDir/$HOST1.ks
    install
    text
    url --url=$YUMREPO
    lang en_US.UTF-8
    keyboard us
    network --onboot yes --device eth0 --bootproto static --ip=$IP --netmask $NETMASK --gateway $GATEWAY --nameserver=$NAMESERVER1 --ipv6 auto
    
    rootpw  --iscrypted \$6\$PD6yBy8dGVjYs.nN\$lfoAkRu8cNLk85iFhsy8XK7RcT5XypJEFjGmW8uLggHeH9JzoUgPIai27R5R3mU29yC9Q7pwYQvMeoMwrb6sj/
    firewall --disabled
    authconfig --enableshadow --passalgo=sha512
    selinux --disabled
    timezone --utc Etc/UTC
    bootloader --location=mbr --driveorder=vda
    
    zerombr
    # Partition table
    clearpart --all
    autopart
    
    repo --name="CentOS" --baseurl=$YUMREPO --cost=100
    
    %packages
    @core
    @server-policy
    @workstation-policy
    
    %post
    
    #Update /etc/resolv.conf
    #sed -i '1s/^/search $DOMAIN\n/' /etc/resolv.conf
    
    #Update /etc/sysconfig/network
    sed -i "/HOSTNAME=/c\HOSTNAME=$HOST1.$DOMAIN" /etc/sysconfig/network
    
    
    #Install salt-minion and change Salt ID
    rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum -y install salt-minion
    sed -i "s/#id:/id: $HOST1.$DOMAIN/" /etc/salt/minion
    sed -i "/#master:/c\master: $SaltMaster" /etc/salt/minion
    
    chkconfig salt-minion on
    service salt-minion restart
    
    %end
    reboot
    
    EOM
    
    echo "Created $TargetDir/$HOST1.ks"
    
    
    
    #####
    # Create %HOST1.sh file
    #####
    echo " "
    cat <<EOM > $TargetDir/$HOST1.sh
    #!/bin/bash
    
    # $HOST1
    
    HOST1=${HOST1}
    YUMREPO=${YUMREPO}
    
    
    virt-install \
    -w bridge=br0 \
    --connect qemu:///system \
    -n \$HOST1 \
    --disk /data2/vm2/\$HOST1.img,size=$DiskSize \
    -r 1024 \
    --vcpus=1 \
    --arch=x86_64 \
    -l \$YUMREPO \
    --os-type=linux \
    --noautoconsole \
    --wait=-1 \
    --os-variant=rhel6 \
    --keymap=en-us \
    --initrd-inject=/root/ks/\$HOST1.ks \
    -x "ks=file:/\$HOST1.ks"
    EOM
    
    chmod 750 $TargetDir/$HOST1.sh
    echo "Created $TargetDir/$HOST1.sh"
    
  4. Most of the variables in the script have default value but you can update most of them by answering prompts from the script.
  5. Most variables are self explanatory. However one variable, YUMREPO, is very important. The variable value http://192.168.12.111/yumrpms/centos$VERSION1/$VERSION1/os/x86_64/ is translated to http://192.168.12.111/yumrpms/centos6.6/6.6/os/x86_64/ when you pick option 1. This URL was used in installing lab-web-tmpl-01 earlier.

Run kickstart-prep.sh to generate files for labweb03

Let's try testing kickstart-prep.sh to prepare to install labweb03.

  1. r2d2: Run /root/bin/kickstart-prep.sh.
  2. Answer the prompts carefully. Remember to use labweb03 as its hostname. Make sure the IP settings you enter are appropriate for your LAN. You may be tempted to just use DHCP assigned IP but that is not really a good option when your goal is to be able to provision multiple KVM Guests automatically.
  3. r2d2: Once you are done with answering the questions, you should have 2 files: /root/ks/labweb03.ks and /root/ks/labweb03.sh.

Test installing labweb03 with the generated .sh file

Let's test running /root/ks/labweb03.sh to install KVM Guest, labweb03. Normally you would run the script with & option. However, because we are testing the file generated by kickstart-prep.sh, we will run labweb03.sh without & option to catch any errors. Open virt-viewer within virt-manager as soon as installation starts to check the status of labweb03. It's quite possible kickstart-prep.sh was missing something and the 2 generated (.ks and .sh) may not be correct. So you want to open the virt-viewer to labweb03 soon and catch any possible error right away. If labweb03 is installed properly and you can remotely manage it with Salt, great. You can move onto using kickstart-prep.sh for provisioning more Linux hosts.

How to provision multiple KVM Guests

With kickstart-prep.sh, you can now create .sh and .ks files needed to provision a KVM Guest in 20 seconds. Let's say you need to provision labweb05 - labweb7 (3 hosts). You'd first run kickstart-prep.sh 3 times to generate files for labweb05 - 07, giving you 3 .sh and 3 .ks files. Once you have the files ready, you can execute the .sh like below. The key advantage is that you don't need to sit there and monitor. To provision 3 new KVM Guests, all that needs to be done, and requiring about 2 minutes of typing are:

  1. Run kickstart-prep.sh 3 times for labweb05, labweb06 and labweb07.
  2. Execute following commands. You can walk away. After some time (3min - 1hr depenging on your hardware), all three Linux hosts will be ready.
[[email protected] ~]# cd /root/ks
[[email protected] ~]# ./labweb05.sh;./labweb06.sh; ./labweb06.sh

Cleanup

Remember to enable firewall on the newly provisioned Linux hosts, which can be accomplished easily with Salt. If a provisioned Linux host gets destroyed (due to bad config etc), you can always rerun the .sh script again to reprovision it.

Boot up KVM Guests

  1. Start all 3 KVM Guests with following command.
  2. virsh start lab02; virsh start lab03; virsh start lab04
    
  3. Wait for a minute for all KVM Guests to finish spinning up. Once they are up, you could ssh log into each one, but you don't need to thanks to Salt.
  4. You can always use virt-manager GUI tool on Mac to start up KVM Guests.

How to delete KVM Guests

  1. To delete a KVM Guest, run following commands. It's actually three separate commands in one line. You connect to the Guest, shut it down, and delete it.
  2.     [[email protected] ks]# virsh --connect qemu:///system destroy lab01; virsh --connect qemu:///system undefine lab01; rm /vm/lab01.img
        
  3. You can always shutdown and delete KVM Guests within virt-manager GUI on your Mac.