Initial setup of SaltStack on CentOS

Posted by paul on 2014.05.16

Initial setup of SaltStack on CentOS

2014.05.16 Fri

I'm falling in love with SaltStack. I've spent some time exploring Puppet in the past. It was not a great experience personally. To get Puppet going, you have to have functioning DNS system. And each machine needs be added into the DNS records. If you have 100 servers, they all need to be reachable via DNS. And when you push out a change via Puppet, the Puppet clients can take as long as 20 minutes to reach out and grab files from the Puppet Master. There's no easy way to make sure all 100 Puppet Clients copy files down and switch out the content of Apache Document directory at the same time. And there's no easy way to generate reports.

Then I learned about SaltStack. My favorite part of Salt is that you can remotely execute commands as root on remote servers. And you don't have to have the servers added to DNS records. In this blog I walk through setting up SaltStack on my CentOS 6.x servers. SaltStack documentation is already GREAT but hopefully this blog can help those looking for an even shorter introduction to using SaltStack on CentOS.

Assumptions

  1. Salt Master is salt01.home.loc at 192.168.11.61. Its running CentOS 6.x.
  2. Salt Minion (or minion) used in this tutorial is test100.home.loc, running CentOS 6.x.
  3. The CentOS server you are working on has no data you need and can be rebooted/reinstalled without causing any issue on your LAN.

Reach Salt Master with hostname or IP

Before installing Salt Master (where you control Salt Minions from), you need to decide how Salt Minions (Salt Clients) will connect to it. Will it be by hostname (preferably with hostname: salt) or IP address?

When you install Salt Minion on a computer and start salt-minion service for the first time (without any configuration), the minion tries to look for the Salt Master using hostname 'salt'. So if your Salt Master can be reached by using hostname 'salt' (note it's not FQDN), you don't have to configure minions at all to start working with Salt. So I recommend having your DNS server set up so that other computers can reach the Salt Master using 'salt'.

If you do not have 'salt' hostname in your DNS servers or don't even have DNS servers in your local environment, you will need to configure minion config file ( /etc/salt/minion ) on each minion and add a line with hostname or IP address pointing to the Salt Master. Simple enough.

In my setup, I have my Salt Master with hostname salt01.home.loc. I also have DNS alias salt.home.loc which points to salt01.home.loc. And resolve.conf on other computers is configured so that they can just use 'salt' in order to reach the Salt Master. However I still configure each /etc/salt/minion so that each minion contacts the Slat Master using IP. This way, even if DNS servers are down for some reason, I can still control the minions from the Salt Master. Using Salt, you can easily modify /etc/salt/minion on all of your minions.

Install packages

Salt Master

On the CentOS 6.x (x86_64) server that will act as the saltMaster, enable epel rpm, if it's not enabled already. In most cases, you need just one Salt Master. The specific URL to get epel-release-6-8.noarch.rpm may need to be changed. Next you install rpms for salt-master and start the salt-master service.

[[email protected] ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install salt-master
[[email protected] ~]# chkconfig salt-master on
[[email protected] ~]# service salt-master start

Salt Minion

On the CentOS 6.x computers that will act as minions, also enable epel rpm. Next you will install salt-minion. Set the salt-minion service to auto start upon system reboot.

[[email protected] ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[[email protected] ~]# yum install salt-minion
[[email protected] ~]# chkconfig salt-minion on

Configure Minion to know about Salt Master

Next, make sure the new minion knows where the Salt Master is. If your local DNS server has hostname 'salt' pointing to the Salt Master, you do not need to configure the new minion. You can simply start the salt-minion service and the minion will attemtpt to contact server with hostname 'salt'. But if you have no local DNS server or prefer using IP (I recommend using IP), edit /etc/salt/minion as shown below.

Here's how you configure a minion to look for a non-'salt' Salt Master.

  1. On your minion, open /etc/salt/minion and look for the lines that look like below.
  2. ...
    # Set the location of the salt master server, if the master server cannot be
    # resolved, then the minion will fail to start.
    #master: salt
    
    # Set whether the minion should connect to the master via IPv6
    #ipv6: False
    ...
    
  3. Add one line: master: 192.168.11.61 as shown below. The IP is of the Salt Master in my environment. Make sure there's no blank space at the beginning of the line but you do need a blank space before the IP address. Python is sensitive about having blank spaces in right places. If not done right, restarting salt-minion service will throw an error. It's a Python thing.
  4. ...
    # Set the location of the salt master server, if the master server cannot be
    # resolved, then the minion will fail to start.
    #master: salt
    master: 192.168.11.61
    
    # Set whether the minion should connect to the master via IPv6
    #ipv6: False
    ...
    
  5. Now start salt-minion service by running 'service salt-minion start'. You can check the log of salt-minion which is at /var/log/salt/minion.
  6. [[email protected] ~]# service salt-minion start
    Starting salt-minion daemon:                               [  OK  ]
    
  7. At this point, Salt Minion is aware of Salt Master but not authorized to get anything from the Salt Master yet.

Configure Salt Master to authorize new Minions

So now your new minion is aware which Salt Master to use and has sent a key to that Salt Master for authorization. Move to the Salt Master to list keys sent from minions by issuing salt-key command.

[[email protected] ~]# salt-key -L
Accepted Keys:
Unaccepted Keys:
test100.home.loc
Rejected Keys:

At this point, only one minion (test100.home.loc) has sent a key and none has been accepted so far. Accept all keys by running following command.

[[email protected] ~]# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
test100.home.loc
Proceed? [n/Y] y
Key for minion test100.home.loc accepted.

Run salt-key -L again to view the state of the keys. Now the key from test100.home.loc is accepted.

[[email protected] ~]# salt-key -L
Accepted Keys:
test100.home.loc
Unaccepted Keys:
Rejected Keys:
[[email protected] ~]#

Test Salt Master can control Salt Minion

Now you are ready to test if Salt Master can actually control the minion. On Salt Master run following command to test. It's not actually sending a 'ping' over the network. This test shows test100.home.loc is reachable by the Salt Master (salt01).

[[email protected] ~]# salt '*' test.ping
test100.home.loc:
    True
[[email protected] ~]#

With * I am targeting all minions as I send out 'test.ping' command. I'm sending a command to all Salt Minions this Salt Master is aware of. There are lots of other flexible ways to target minions. I could've issued following commands. You can target specific groups of minions so you can use '*100.home.loc' to target the new minion.

[[email protected] ~]# salt '*.home.loc' test.ping
test100.home.loc:
    True
[[email protected] ~]# salt '*100.home.loc' test.ping
test100.home.loc:
    True

Here's why I like SaltStack so much already. Let's say I want to find out the HD usage on the my servers that are minions. Here's the command, one line. I can tell immediately my minion, test100, has a relatively empty HD. Note 'disk.percent' is what's called module.function in Salt.

[[email protected] ~]# salt '*' disk.percent
test100.home.loc:
    ----------
    /:
        16%
    /boot:
        13%
    /dev/shm:
        0%

What if you had 20 servers of various Linux flavors (and even some Windows) and you had to find out disk usage. You could've written a shell script (or 2 or more depending on what OSes you have) and figure out how to copy them to remote Linux servers and run them as 'root' and somehow get the results back into one place. Some sys admins do not have the scripting skill to get this done in a relatively short time. You will need to write the scripts and test them. MAYBE your typing is fast enough that you can shell into each server and paste/run a command and scp the output to a central server. I used to do something like it.

But with Salt running on your computers, you can simply run a short command on the Salt Master once. With just one short command you will get the needed result in a few seconds. I think this alone justifies getting Salt into your environment.

Remote execute with Salt

I can find out lots of info of remote machines without having to shell into each server. I haven't even gotten to the part where I modify the remote systems but already SaltStack is proving its usefulness.

Salt comes with a ton of functions. You can find out what are available on your minions by running following command. Use '| less' to examine the output. There are hundreds of lines in the return. They are also documented at http://docs.saltstack.com/en/latest/ref/modules/all/index.html

[[email protected] ~]# salt '*' sys.doc | less

The cmd module contains functions, one of which is cmd.run function. This cmd.run (module.function) let's you execute shell command on remote servers as root. Here's how you can use cmd.run. Basically the command runs 'ls /etc' on all minions and returns the result.

[[email protected] ~]# salt '*' cmd.run "ls /etc" | less
...
test100.home.loc:
    ConsoleKit
    DIR_COLORS
    DIR_COLORS.256color
    DIR_COLORS.lightbgcolor
    NetworkManager
    Trolltech.conf
    X11
    abrt
    acpi
    adjtime
    aliases
    aliases.db
    alsa
    alternatives
    anacrontab
    asound.conf
    at.deny
    audisp
    audit
    auto.master
    auto.misc
    auto.net
    auto.smb
 ...

Find a minion with a specific MAC address

Included in network module is network.interfaces module.function. It returns info of all interfaces of a minion, including IP and MAC addresses, etc. If you ever needed to track down MAC address on one server out of hundreds of minions (minions), you can simply run following command. Let's assume the MAC address you are hunting for is 52:54:01:04:0e:08. You could possibly hunt it down using management feature of a switch. But with SaltStack, it's mindlessly simple.

[[email protected] ~]# salt '*' network.interfaces | grep -B 10            52:54:01:04:0e:08
test100.home.loc:
    ----------
    eth0:
        ----------
        hwaddr:
            52:54:01:04:0e:08

Note ifconfig on linux returns 52:54:01:04:0E:08, with upper case E. But salt '*' network.interfaces returns 52:54:00:54:0e:58 with lowercase e. Despite this minor issue, the command is still very useful.

With these basics covered, I will cover Salt States in next blog. Salt States is the Salt component for configuration management.