Using KVM, Kickstart, SaltStack

To provision multiple CentOS rapidly

Posted by paul on 2014.06.06

Using KVM/Kickstart/SaltStack to provision multiple CentOS rapidly

2014.06.06 Fri

KVM is great for creating a multi-server environment with just one physical server. And it's free. But installing a KVM Guest running CentOS had been a manual task for me for awhile. I finally got around to automating it. By combining KVM/Kickstart/SaltStack, I was able to automate the whole installation process and also have the ability to start remote controlling the new KVM Guests. You need to spend just a few minutes creating a script and some Kickstart files. You then execute the script and walk away. After about 30 minutes or a few hours (depending on how many you create and the hardware of the KVM Host), you have multiple KVM Guests with CentOS. Best of all, you can start remote controlling them all with SaltStack.

Assumptions

  1. All computers run CentOS 6.5 x86_64. All CentOS 6.x and even 5.x should work but this tutorial was written using 6.5
  2. r2d2.base.loc is the KVM host, physical box running CentOS 6.5 x86_64.
  3. My Wireless Router's DHCP server gives out following: IP 192.168.12.xxx Netmask 255.255.255.0 Gateway 192.168.12.1. DNS server is 192.168.12.9.
  4. You have a working local Yumrepo server. If not, follow my tutorial here to set one up. In this tutorial, it's assumed to be running on 192.168.12.111.
  5. Each KVM Guest being installed will get a static IP. I will use the following for this tutorial.
    1. lab01: 192.168.12.151
    2. lab02: 192.168.12.152
    3. lab03: 192.168.12.153

Find a Kickstart file

In order to install a CentOS computer without having to reply to any prompts manually, you need to have a Kickstart file. Whenever you install a CentOS computer you will end up with /root/anaconda-ks.cfg. This file is a log of things that were done/installed by Anaconda, installer of CentOS. You can use this template to create a Kickstart file by renaming the file as somename.ks. You do need to add some more options for more control in the install process.

  1. On your KVM Host (r2d2), create directory /root/ks/ to hold Kickstart files.
  2. I recommend creating a new KVM Guest with minimal options to get a fresh anaconda-ks.cfg.
  3. Log into r2d2 (ssh -Y [email protected]), open virt-manager and install a new KVM Guest (call it lab-tmpl-01) running CentOS 6.5 with the default options you want on all future KVM Guests running CentOS 6. Remember that the root password you enter in this install will be used in all following KVM Guests installed using this Kickstart file.
  4. When you install a CentOS 6.5 manually, you need to edit a line in ifcfg-eth0 to get it online.
  5. After rebooting lab-tmpl-01, log in and edit /etc/sysconfig/network-scripts/ifcfg-eth0 and change following line. Then reestart network service. If you have DHCP server on your LAN, lab-tmpl-01 should be online now.
  6. #OLD
    ONBOOT=no
    
    #NEW
    ONBOOT=yes
    
  7. After lab-tmpl-01 is installed and connected to the LAN, copy lab-tmpl-01:/root/anaconda-ks.cfg --> r2d2:/root/ks/lab-tmpl-01.ks. Note the file extension is changed.
  8. On r2d2, the Kickstart file /root/ks/lab-tmpl-01.ks will look like something like this, before customization.
  9. # Kickstart file automatically generated by anaconda.
    
    #version=DEVEL
    install
    cdrom
    lang en_US.UTF-8
    keyboard us
    network --onboot no --device eth0 --bootproto dhcp --noipv6
    rootpw  --iscrypted $6$Bkt8t7sdFH7wgWXy$HOs6ka9JJ.8Racg/cEhBsuD/UOhA8nirk2hGxIGc190XdRl/kTjv/WOcPDv1
    firewall --service=ssh
    authconfig --enableshadow --passalgo=sha512
    selinux --enforcing
    timezone --utc America/New_York
    bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"
    # The following is the partition information you requested
    # Note that any partitions you deleted are not expressed
    # here so unless you clear all partitions first, this is
    # not guaranteed to work
    #clearpart --all --drives=sda
    #volgroup VolGroup --pesize=4096 pv.008002
    #logvol / --fstype=ext4 --name=lv_root --vgname=VolGroup --grow --size=1024 --maxsize=51200
    #logvol swap --name=lv_swap --vgname=VolGroup --grow --size=1843 --maxsize=1843
    
    #part /boot --fstype=ext4 --size=500
    #part pv.008002 --grow --size=1
    
    
    repo --name="CentOS"  --baseurl=cdrom:sr0 --cost=100
    
    %packages
    @base
    @console-internet
    @core
    @debugging
    @directory-client
    @hardware-monitoring
    @java-platform
    @large-systems
    @network-file-system-client
    @performance
    @perl-runtime
    @server-platform
    @server-policy
    @workstation-policy
    pax
    oddjob
    sgpio
    device-mapper-persistent-data
    samba-winbind
    certmonger
    pam_krb5
    krb5-workstation
    perl-DBD-SQLite
    %end
    

Edit Kickstart file to install lab01

  1. On r2d2, copy /root/ks/lab-tmpl-01.ks --> /root/ks/lab01.ks.
  2. You will edit /root/ks/lab01.ks so that it looks like below.
  3. # Kickstart file automatically generated by anaconda.
    
    #version=DEVEL
    install
    text
    url --url=http://192.168.12.111/yumrpms/centos/6.5/os/x86_64
    lang en_US.UTF-8
    keyboard us
    network --onboot yes --device eth0 --bootproto static --ip=192.168.12.151 --netmask 255.255.255.0 --gateway 192.168.12.1 --nameserver=192.168.12.11 --ipv6 auto
    rootpw  --iscrypted $6$e/K6hYbvX7DTD1eh$UHf6zUBCVQjJvNExHPob3ibKZdq5tg.NczGL0WAGA68SMCVX29jGvUCxGe8c9972ijPY9cJdneQBffD5yzxtL/
    firewall --disabled
    authconfig --enableshadow --passalgo=sha512
    selinux --disabled
    timezone --utc America/Los_Angeles
    bootloader --location=mbr --driveorder=vda
    
    zerombr
    # Partition table
    clearpart --all
    autopart
    
    repo --name="CentOS"  --baseurl=http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 --cost=100
    reboot
    #poweroff
    #cmdline
    
    %packages
    @base
    @console-internet
    @core
    @debugging
    @directory-client
    @hardware-monitoring
    @java-platform
    @large-systems
    @network-file-system-client
    @performance
    @perl-runtime
    @server-platform
    @server-policy
    @workstation-policy
    pax
    oddjob
    sgpio
    device-mapper-persistent-data
    samba-winbind
    certmonger
    pam_krb5
    krb5-workstation
    perl-DBD-SQLite
    %end
    
    
    1. Add: text. For using text version of anaconda.
    2. #OLD
      ...
      #version=DEVEL
      install
      ...
      
      
      #NEW
      ...
      #version=DEVEL
      install
      text
      ...
      
    3. Replace line: cdrom. To use the install RPMs on your local Yumrepo server.
    4. #OLD
      ...
      cdrom
      ...
      
      
      #NEW
      ...
      url --url=http://192.168.12.111/yumrpms/centos/6.5/os/x86_64
      ...
      
    5. Update line: network --onboot .... With this line, you are giving a static IP, Netmask, Gateway, and DNS server to lab01.
    6. #OLD
      ...
      network --onboot no --device eth0 --bootproto dhcp --noipv6
      ...
      
      
      #NEW
      ...
      network --onboot yes --device eth0 --bootproto static --ip=192.168.12.151 --netmask 255.255.255.0 --gateway 192.168.12.1 --nameserver=192.168.12.11 --ipv6 auto
      ...
      
    7. Update following 5 lines. You don't need to update the first 3 lines but you will need to update the last 2 lines. Since I'm installing lab computers, I disable firewall/selinux. Note 'sda' is updated to 'vda' since it's your first virtual drive. And you want to use your local Yumrepo server.
    8. #OLD
      ...
      firewall --service=ssh
      ...
      selinux --enforcing
      timezone --utc America/New_York
      bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"
      ...
      repo --name="CentOS"  --baseurl=cdrom:sr0 --cost=100
      ...
      
      #NEW
      
      ...
      firewall --service=disabled
      ...
      selinux --disabled
      timezone --utc America/Los_Angeles
      bootloader --location=mbr --driveorder=vda
      ...
      repo --name="CentOS"  --baseurl=http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 --cost=100
      
    9. Add following 2 lines. 'Reboot' forces the newly installed KVM Guest to reboot automatically after the installation is done. 'Poweroff' shuts down the new KVM Guest but we are not using it here so it's commented out. You can use only one of the 2 options at a time.
    10. reboot
      #poweroff
      

Prepare virt-lab01.sh

Now you are ready to issue virt-install command on r2d2 (your KVM Host server) to install your first KVM Guest, lab01. Without using Kickstart file, you would need to provide manual inputs. But because you will use the Kickstart file (/root/ks/lab01.ks), you won't need to sit there and click on options.

  1. On r2d2, create file /root/ks/virt-lab01.sh with following. You can save the file anywhere but I keep it with the Kickstart files.
  2. #!/bin/bash
    
    virt-install \
    -w bridge=br0 \
    --connect qemu:///system \
    -n lab01 \
    --disk /vm/lab01.img,size=19 \
    -r 1024 \
    --vcpus=1 \
    --arch=x86_64 \
    -l http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 \
    --os-type=linux \
    --os-variant=rhel6 \
    --keymap=en-us \
    --initrd-inject=/root/ks/lab01.ks \
    -x "ks=file:/lab01.ks"
    
    1. -w bridge=br0 --> This sets the KVM Guest to use the bridge network of KVM Host, ensuring the KVM Guest has full inbound/outbound network access.
    2. -n lab01 --> Names the KVM Guest as lab01, with the KVM. Try to use same name you use for the hostname of the KVM Guest.
    3. --disk /vm/lab01.img,size=19 --> creates virtual disk image at /vm/lab01.img and sets the max size to 19GB.
    4. -r 1024 --> Sets the memory to 1024MB.
    5. --vcpus=1 --> sets to use 1 virtual CPU.
    6. --arch=x86_64 --> sets to use 64bit architecture.
    7. -l http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 --> Source rpms. This is on the local Yumrepo server.
    8. --keymap=en-us --> sets the keyboard to US English mapping.
    9. --initrd-inject=/root/ks/lab01.ks --> tells virt-install command where to look for lab01.ks, the Kickstart file. It's the actual file path of lab01.ks.
    10. -x "ks=file:/lab01.ks --> tells virt-install command to use lab01.ks as the Kickstart file. Note the / at beginning of /lab01.ks. That's not a typo.
  3. Save /root/ks/virt-lab01.sh and make it executable.

Run virt-lab01.sh to install lab01

  1. Log into KVM Host (r2d2) as root (ssh -Y [email protected]) and run /root/ks/virt-lab01.sh.
  2. You will see following lines scroll by in your Terminal. When the last line shows, it seems it's hanging as it's still there even when you hit 'Enter' key. That will be fixed later.
  3. [[email protected] ks]# ./virt-lab01.sh
    Starting install...
    Retrieving file .treeinfo...                        |  728 B     00:00 ...
    Retrieving file vmlinuz...                          | 7.9 MB     00:00 ...
    Retrieving file initrd.img...                       |  64 MB     00:06 ...
    Creating storage file lab01.img                     |  19 GB     00:00
    Creating domain...                                  |    0 B     00:00
    
    (virt-viewer:24954): gtk-vnc-WARNING **: unknown keycodes `empty_aliases(qwerty)', please report to gtk-vnc-devel
    
    
    
  4. You will also see a virt-viewer window open showing the virtual screen of the KVM Guest being installed. After about 10-30 min (depending on the KVM Host), the install will be completed and the newly installed KVM Guest will be automatically rebooted. It will have a working network with IP address (in this case 192.168.12.151). You didn't have to edit ifcfg-eth0 manually to get the CentOS KVM online.
  5. Once lab01 has rebooted, open another Terminal windows and your Mac and try ssh log into 192.168.12.151 as root. Use the root password you used when you were installing lab-tmpl-01 earlier. You should be able to log in.
  6. However, in the original Terminal you are stuck with following line. You have to use Ctrl+c to get out of the frozen state.
  7. (virt-viewer:29092): gtk-vnc-WARNING **: unknown keycodes `empty_aliases(qwerty)', please report to gtk-vnc-devel
    
    
  8. And when you do Ctrl+c out, the virt-viewer of lab01 is forcibly closed.
  9. Let's update the Kickstart file and also the virt-lab01.sh to get rid of hanging Terminal. Also some more enhancements will be added.

Add post install options to Kickstart file, lab01.

So now you have all necessary pieces for installing a KVM Guest using Kickstart. Let's update the Kickstart file lab01.ks, including adding some lines to install Salt-Minion. Installing Salt-Minion using Kickstart adds great flexibility and allows you to immediately start working with all KVM Guests. You don't ever have to log into each KVM Guest individually. After adding these updates, each KVM Guest will have a unique hostname (file: /etc/sysconfig/network). If you don't have Salt Master at hostname (salt) running in your LAN, you can skip lines 2 - 3 & 8 - 13. But using SaltStack is highly recommended.

  1. Open /root/ks/lab01.ks and add following lines BEFORE %end.
  2. %post
    # 1 Update /etc/resolv.conf
    sed -i '1s/^/search base.loc\n/' /etc/resolv.conf
    
    # 2 Update /etc/sysconfig/network
    sed -i "/HOSTNAME=/c\HOSTNAME=lab01.base.loc" /etc/sysconfig/network
    
    # 3 Install salt-minion and change Salt ID
    rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum -y install salt-minion
    sed -i "s/#id:/id: lab01.base.loc/" /etc/salt/minion
    
    #4 Update Master Server name. Change saltserver.base.loc to match yours. If your Salt Master is reachable with 'salt', skip this.
    sed -i "/#master:/c\master: saltserver.base.loc" /etc/salt/minion
    
    #5 Start salt-minion service
    chkconfig salt-minion on
    service salt-minion restart
    
    1. #1. /etc/resolv.conf --> is updated so lab01 can reach Salt Master by just looking for 'salt'. Change "base.loc" to match your environment.
    2. #2. /etc/sysconfig/network --> is updated so that it's easy to identify different CentOS computers when logging in.
    3. #3. EPEL repo is added and Salt-Minion is installed. Also Salt ID is specified to match the hostname.
    4. #4. saltserver.base.loc --> Specify hostname of the Salt Master to look for. Change salt.server.base.loc to match yours. If your Salt Master is reachable via 'salt', you can skip this. Salt Minion looks for 'salt' by default.
  3. At this point /root/ks/lab01.ks should look like below:
  4. # Kickstart file automatically generated by anaconda.
    
    #version=DEVEL
    install
    text
    url --url=http://192.168.12.111/yumrpms/centos/6.5/os/x86_64
    lang en_US.UTF-8
    keyboard us
    network --onboot yes --device eth0 --bootproto static --ip=192.168.12.151 --netmask 255.255.255.0 --gateway 192.168.12.1 --nameserver=192.168.12.11 --ipv6 auto
    rootpw  --iscrypted $6$e/K6hYbvX7DTD1eh$UHf6zUBCVQjJvNExHPob3ibKZdq5tg.NczGL0WAGA68SMCVX29jGvUCxGe8c9972ijPY9cJdneQBffD5yzxtL/
    firewall --disabled
    authconfig --enableshadow --passalgo=sha512
    selinux --disabled
    timezone --utc America/Los_Angeles
    bootloader --location=mbr --driveorder=vda
    
    zerombr
    # Partition table
    clearpart --all
    autopart
    
    repo --name="CentOS"  --baseurl=http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 --cost=100
    #reboot
    poweroff
    #cmdline
    
    %packages
    @base
    @console-internet
    @core
    @debugging
    @directory-client
    @hardware-monitoring
    @java-platform
    @large-systems
    @network-file-system-client
    @performance
    @perl-runtime
    @server-platform
    @server-policy
    @workstation-policy
    pax
    oddjob
    sgpio
    device-mapper-persistent-data
    samba-winbind
    certmonger
    pam_krb5
    krb5-workstation
    perl-DBD-SQLite
    
    
    %post
    #Update /etc/resolv.conf
    sed -i '1s/^/search base.loc\n/' /etc/resolv.conf
    
    #Update /etc/sysconfig/network
    sed -i "/HOSTNAME=/c\HOSTNAME=lab01.base.loc" /etc/sysconfig/network
    
    #Install salt-minion and change Salt ID
    rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    yum -y install salt-minion
    sed -i "s/#id:/id: lab01.base.loc/" /etc/salt/minion
    
    #Update Master Server name. Change saltserver.base.loc to match yours. If your Salt Master is reachable with 'salt', skip this.
    sed -i "/#master:/c\master: saltserver.base.loc" /etc/salt/minion
    
    #Start salt-minion service
    chkconfig salt-minion on
    service salt-minion restart
    
    %end
    

How to delete KVM Guests

  1. To delete a KVM Guest, run following commands. It's actually three separate commands in one line.
  2. [[email protected] ks]# virsh --connect qemu:///system destroy lab01; virsh --connect qemu:///system undefine lab01; rm /vm/lab01.img
    

Prepare multiple Kickstart files for installing multiple KVM Guests

  1. Let's prepare to install 3 KVM Guests using just one script.
  2. First you need to prepare 3 more Kickstart files, to be named lab02.ks, lab03.ks & lab04.ks.
  3. [[email protected] ks]# cp lab01.ks lab02.ks
    [[email protected] ks]# cp lab01.ks lab03.ks
    [[email protected] ks]# cp lab01.ks lab04.ks
    
  4. In each Kickstart file, 3 places need to be updated. The changes are needed to issue unique IP/hostname to each KVM Guest. Use sed command to quickly update all 3 files.
  5. Update lab02.ks
  6. sed -i 's/151/152/' lab02.ks
    sed -i 's/lab01.base.loc/lab02.base.loc/' lab02.ks
    
  7. Update lab03.ks
  8.     sed -i 's/151/153/' lab03.ks
        sed -i 's/lab01.base.loc/lab03.base.loc/' lab03.ks
    
  9. Update lab04.ks
  10.     sed -i 's/151/154/' lab04.ks
        sed -i 's/lab01.base.loc/lab04.base.loc/' lab04.ks
    

Prepare virt-2-3-4.sh

  1. Let's create a script to run 3 virt-install commands, installing lab02, lab03 and lab04.
  2. Copy /root/ks/virt-lab01.sh to /root/ks/virt-2-3-4.sh
  3. Edit /root/ks/virt-2-3-4.sh so it looks like below.
  4. #!/bin/bash
    
    #lab02
    virt-install \
    -w bridge=br0 \
    --connect qemu:///system \
    -n lab02 \
    --disk /vm/lab02.img,size=19 \
    -r 1024 \
    --vcpus=1 \
    --arch=x86_64 \
    -l http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 \
    --os-type=linux \
    --os-variant=rhel6 \
    --noautoconsole \
    --keymap=en-us \
    --initrd-inject=/root/ks/lab02.ks \
    -x "ks=file:/lab02.ks" &
    
    
    #lab03
    virt-install \
    -w bridge=br0 \
    --connect qemu:///system \
    -n lab03 \
    --disk /vm/lab03.img,size=19 \
    -r 1024 \
    --vcpus=1 \
    --arch=x86_64 \
    -l http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 \
    --os-type=linux \
    --os-variant=rhel6 \
    --noautoconsole \
    --keymap=en-us \
    --initrd-inject=/root/ks/lab03.ks \
    -x "ks=file:/lab03.ks" &
    
    sleep 30m
    
    #lab04
    virt-install \
    -w bridge=br0 \
    --connect qemu:///system \
    -n lab04 \
    --disk /vm/lab04.img,size=19 \
    -r 1024 \
    --vcpus=1 \
    --arch=x86_64 \
    -l http://192.168.12.111/yumrpms/centos/6.5/os/x86_64 \
    --os-type=linux \
    --os-variant=rhel6 \
    --noautoconsole \
    --keymap=en-us \
    --initrd-inject=/root/ks/lab04.ks \
    -x "ks=file:/lab04.ks" &
    
    wait
    exit 0
    
  5. --noautoconsole --> added in each virt-install command so that virt-install doesn't automatically open a virt-viewer window. When installing multiple KVM Guests, you want to avoid opening a virt-viewer for each new KVM Guest.
  6. sleep 30m --> lab02 and lab03 will be installed simultaneously as soon as virt-2-3-4.sh is executed. But in order to avoid overwhelming the KVM Host, install of lab04 will wait 30 minutes.
  7. Note when you go from lab02 to lab03, all four instances of 'lab02' should be changed to 'lab03'. Same with 'lab04'. Important not to miss these.
  8. & --> & at the end of each virt-install command ensures the virt-install command runs in the background.
  9. wait --> this last line is added so that the script /root/ks/virt-2-3-4.sh doesn't act as if it's frozen after the last virt-install command is issued. Virt-1-2-3.sh will exit gracefully once installation of lab04 starts.

Run virt-2-3-4.sh.

  1. Make virt-2-3-4.sh executable and run it.
  2. Here's what you will see. The last line will show up at the beginning of the installation of lab04. Note no virt-viewer window will open.
  3. [[email protected] ks]# ./lab1-2-3.sh
    
    Starting install...
    
    Starting install...
    Retrieving file .treeinfo...                   |  728 B     00:00 ...
    Retrieving file .treeinfo...                   |  728 B     00:00 ...
    Retrieving file vmlinuz...                     | 7.9 MB     00:01 ...
    Retrieving file vmlinuz...                     | 7.9 MB     00:01 ...
    Retrieving file initrd.img...                  |  64 MB     00:07 ...
    Creating storage file lab01.img                |  19 GB     00:00
    Retrieving file initrd.img...                  |  64 MB     00:08 ...
    Creating storage file lab02.img                |  19 GB     00:00
    Creating domain...                             |    0 B     00:00
    Domain installation still in progress. You can reconnect to
    the console to complete the installation process.
    Creating domain...                             |    0 B     00:00
    Domain installation still in progress. You can reconnect to
    the console to complete the installation process.
    Done installing KVM Guests
    
    Starting install...
    Retrieving file .treeinfo...                   |  728 B     00:00 ...
    Retrieving file vmlinuz...                     | 7.9 MB     00:00 ...
    Retrieving file initrd.img...                  |  64 MB     00:04 ...
    Creating storage file lab03.img                |  19 GB     00:00
    Creating domain...                             |    0 B     00:00
    Domain installation still in progress. You can reconnect to
    the console to complete the installation process.
    [[email protected] ks]#
    
  4. You will also notice all KVM Guests remain shut down instead of just rebooting.

Boot up KVM Guests

  1. Start all 3 KVM Guests with following command.
  2. virsh start lab02; virsh start lab03; virsh start lab04
    
  3. Wait for a minute for all KVM Guests to finish spinning up. Once they are up, you could ssh log into each one, but you don't need to thanks to Salt.

Salt-Minion

  1. Now you could ssh log into each KVM Guest from your Mac's Terminal to start configuring them. It's not that big of a deal with 3 KVM Guests, BUT it definitely would be if you had just installed 50 KVM Guests. And this is why Salt-Minion was installed early on using Kickstart.
  2. Log into your Salt Master server and run 'salt-key -L'.
  3. [[email protected] ~]# salt-key -L
    Accepted Keys:
    ...
    Unaccepted Keys:
    lab02.base.loc
    lab03.base.loc
    lab04.base.loc
    Rejected Keys:
    [[email protected] ~]#
    
  4. You can see all three new KVM Guests have sent keys to be signed by the Salt Master. Accept all keys from lab02, lab03, & lab04.
  5. [[email protected] ~]# salt-key -A
    The following keys are going to be accepted:
    Unaccepted Keys:
    lab02.base.loc
    lab03.base.loc
    lab04.base.loc
    Proceed? [n/Y] y
    Key for minion lab02.base.loc accepted.
    Key for minion lab03.base.loc accepted.
    Key for minion lab04.base.loc accepted.
    [[email protected] ~]#
    
  6. Test all are remote controllable from the Salt Master. Run following command on your Salt Master. You should get following result.
  7. [[email protected] ~]# salt "lab*" test.ping
    lab01.base.loc:
        True
    lab02.base.loc:
        True
    lab03.base.loc:
        True
    [[email protected] ~]#
    
  8. Check date/time on all 3 servers to make sure Time zone was changed from East US to West US or any other location you picked in Kickstart files. Again, you run 1 command on Salt Master once to get result from all servers.
  9. [[email protected] ~]# salt "lab*" cmd.run "date"
    lab01.base.loc:
        Wed Jun 11 09:43:48 PDT 2014
    lab02.base.loc:
        Wed Jun 11 09:43:48 PDT 2014
    lab03.base.loc:
        Wed Jun 11 09:43:48 PDT 2014
    [[email protected] ~]#
    

That's it. Now you have all the necessary pieces to install dozens of KVM Guests with just a few minutes of typing.