I'm falling in love with SaltStack. I've spent some time exploring Puppet in the past. It was not a great experience personally. To get Puppet going, you have to have functioning DNS system. And each machine needs be added into the DNS records. If you have 100 servers, they all need to be reachable via DNS. And when you push out a change via Puppet, the Puppet clients can take as long as 20 minutes to reach out and grab files from the Puppet Master. There's no easy way to make sure all 100 Puppet Clients copy files down and switch out the content of Apache Document directory at the same time. And there's no easy way to generate reports.
Then I learned about SaltStack. My favorite part of Salt is that you can remotely execute commands as root on remote servers. And you don't have to have the servers added to DNS records. In this blog I walk through setting up SaltStack on my CentOS 6.x servers. SaltStack documentation is already GREAT but hopefully this blog can help those looking for an even shorter introduction to using SaltStack on CentOS.
- Salt Master is salt01.home.loc at 192.168.11.61. Its running CentOS 6.x.
- Salt Minion (or minion) used in this tutorial is test100.home.loc, running CentOS 6.x.
- The CentOS server you are working on has no data you need and can be rebooted/reinstalled without causing any issue on your LAN.
Reach Salt Master with hostname or IP
Before installing Salt Master (where you control Salt Minions from), you need to decide how Salt Minions (Salt Clients) will connect to it. Will it be by hostname (preferably with hostname: salt) or IP address?
When you install Salt Minion on a computer and start salt-minion service for the first time (without any configuration), the minion tries to look for the Salt Master using hostname 'salt'. So if your Salt Master can be reached by using hostname 'salt' (note it's not FQDN), you don't have to configure minions at all to start working with Salt. So I recommend having your DNS server set up so that other computers can reach the Salt Master using 'salt'.
If you do not have 'salt' hostname in your DNS servers or don't even have DNS servers in your local environment, you will need to configure minion config file ( /etc/salt/minion ) on each minion and add a line with hostname or IP address pointing to the Salt Master. Simple enough.
In my setup, I have my Salt Master with hostname salt01.home.loc. I also have DNS alias salt.home.loc which points to salt01.home.loc. And resolve.conf on other computers is configured so that they can just use 'salt' in order to reach the Salt Master. However I still configure each /etc/salt/minion so that each minion contacts the Slat Master using IP. This way, even if DNS servers are down for some reason, I can still control the minions from the Salt Master. Using Salt, you can easily modify /etc/salt/minion on all of your minions.
On the CentOS 6.x (x86_64) server that will act as the saltMaster, enable epel rpm, if it's not enabled already. In most cases, you need just one Salt Master. The specific URL to get epel-release-6-8.noarch.rpm may need to be changed. Next you install rpms for salt-master and start the salt-master service.
[[email protected] ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [[email protected] ~]# yum install salt-master [[email protected] ~]# chkconfig salt-master on [[email protected] ~]# service salt-master start
On the CentOS 6.x computers that will act as minions, also enable epel rpm. Next you will install salt-minion. Set the salt-minion service to auto start upon system reboot.
[[email protected] ~]# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [[email protected] ~]# yum install salt-minion [[email protected] ~]# chkconfig salt-minion on
Configure Minion to know about Salt Master
Next, make sure the new minion knows where the Salt Master is. If your local DNS server has hostname 'salt' pointing to the Salt Master, you do not need to configure the new minion. You can simply start the salt-minion service and the minion will attemtpt to contact server with hostname 'salt'. But if you have no local DNS server or prefer using IP (I recommend using IP), edit /etc/salt/minion as shown below.
Here's how you configure a minion to look for a non-'salt' Salt Master.
- On your minion, open /etc/salt/minion and look for the lines that look like below.
- Add one line: master: 192.168.11.61 as shown below. The IP is of the Salt Master in my environment. Make sure there's no blank space at the beginning of the line but you do need a blank space before the IP address. Python is sensitive about having blank spaces in right places. If not done right, restarting salt-minion service will throw an error. It's a Python thing.
- Now start salt-minion service by running 'service salt-minion start'. You can check the log of salt-minion which is at /var/log/salt/minion.
- At this point, Salt Minion is aware of Salt Master but not authorized to get anything from the Salt Master yet.
... # Set the location of the salt master server, if the master server cannot be # resolved, then the minion will fail to start. #master: salt # Set whether the minion should connect to the master via IPv6 #ipv6: False ...
... # Set the location of the salt master server, if the master server cannot be # resolved, then the minion will fail to start. #master: salt master: 192.168.11.61 # Set whether the minion should connect to the master via IPv6 #ipv6: False ...
[[email protected] ~]# service salt-minion start Starting salt-minion daemon: [ OK ]
Configure Salt Master to authorize new Minions
So now your new minion is aware which Salt Master to use and has sent a key to that Salt Master for authorization. Move to the Salt Master to list keys sent from minions by issuing salt-key command.
[[email protected] ~]# salt-key -L Accepted Keys: Unaccepted Keys: test100.home.loc Rejected Keys:
At this point, only one minion (test100.home.loc) has sent a key and none has been accepted so far. Accept all keys by running following command.
[[email protected] ~]# salt-key -A The following keys are going to be accepted: Unaccepted Keys: test100.home.loc Proceed? [n/Y] y Key for minion test100.home.loc accepted.
Run salt-key -L again to view the state of the keys. Now the key from test100.home.loc is accepted.
[[email protected] ~]# salt-key -L Accepted Keys: test100.home.loc Unaccepted Keys: Rejected Keys: [[email protected] ~]#
Test Salt Master can control Salt Minion
Now you are ready to test if Salt Master can actually control the minion. On Salt Master run following command to test. It's not actually sending a 'ping' over the network. This test shows test100.home.loc is reachable by the Salt Master (salt01).
[[email protected] ~]# salt '*' test.ping test100.home.loc: True [[email protected] ~]#
With * I am targeting all minions as I send out 'test.ping' command. I'm sending a command to all Salt Minions this Salt Master is aware of. There are lots of other flexible ways to target minions. I could've issued following commands. You can target specific groups of minions so you can use '*100.home.loc' to target the new minion.
[[email protected] ~]# salt '*.home.loc' test.ping test100.home.loc: True [[email protected] ~]# salt '*100.home.loc' test.ping test100.home.loc: True
Here's why I like SaltStack so much already. Let's say I want to find out the HD usage on the my servers that are minions. Here's the command, one line. I can tell immediately my minion, test100, has a relatively empty HD. Note 'disk.percent' is what's called module.function in Salt.
[[email protected] ~]# salt '*' disk.percent test100.home.loc: ---------- /: 16% /boot: 13% /dev/shm: 0%
What if you had 20 servers of various Linux flavors (and even some Windows) and you had to find out disk usage. You could've written a shell script (or 2 or more depending on what OSes you have) and figure out how to copy them to remote Linux servers and run them as 'root' and somehow get the results back into one place. Some sys admins do not have the scripting skill to get this done in a relatively short time. You will need to write the scripts and test them. MAYBE your typing is fast enough that you can shell into each server and paste/run a command and scp the output to a central server. I used to do something like it.
But with Salt running on your computers, you can simply run a short command on the Salt Master once. With just one short command you will get the needed result in a few seconds. I think this alone justifies getting Salt into your environment.
Remote execute with Salt
I can find out lots of info of remote machines without having to shell into each server. I haven't even gotten to the part where I modify the remote systems but already SaltStack is proving its usefulness.
Salt comes with a ton of functions. You can find out what are available on your minions by running following command. Use '| less' to examine the output. There are hundreds of lines in the return. They are also documented at http://docs.saltstack.com/en/latest/ref/modules/all/index.html
[[email protected] ~]# salt '*' sys.doc | less
The cmd module contains functions, one of which is cmd.run function. This cmd.run (module.function) let's you execute shell command on remote servers as root. Here's how you can use cmd.run. Basically the command runs 'ls /etc' on all minions and returns the result.
[[email protected] ~]# salt '*' cmd.run "ls /etc" | less ... test100.home.loc: ConsoleKit DIR_COLORS DIR_COLORS.256color DIR_COLORS.lightbgcolor NetworkManager Trolltech.conf X11 abrt acpi adjtime aliases aliases.db alsa alternatives anacrontab asound.conf at.deny audisp audit auto.master auto.misc auto.net auto.smb ...
Find a minion with a specific MAC address
Included in network module is network.interfaces module.function. It returns info of all interfaces of a minion, including IP and MAC addresses, etc. If you ever needed to track down MAC address on one server out of hundreds of minions (minions), you can simply run following command. Let's assume the MAC address you are hunting for is 52:54:01:04:0e:08. You could possibly hunt it down using management feature of a switch. But with SaltStack, it's mindlessly simple.
[[email protected] ~]# salt '*' network.interfaces | grep -B 10 52:54:01:04:0e:08 test100.home.loc: ---------- eth0: ---------- hwaddr: 52:54:01:04:0e:08
Note ifconfig on linux returns 52:54:01:04:0E:08, with upper case E. But salt '*' network.interfaces returns 52:54:00:54:0e:58 with lowercase e. Despite this minor issue, the command is still very useful.
With these basics covered, I will cover Salt States in next blog. Salt States is the Salt component for configuration management.