Wanted to find a way to view/search logs (mainly from Apache and OS logs) from multiple sources in one central location. I decided to try out Splunk since that's what most DevOps teams seem to be using for log analysis. Installing Splunk and setting it up is simple enough but for first timers, wading through the Splunk documentation can be a bit of challenge. Hence this blog.
- Using CentOS 6.5 64bit for both Splunk Enterprise (aka indexer)and splunkforwarder (where logs come from). I will start using Splunk Enterprise but I am going to change the license to the free version (which limits amount of log data it can digest to 500MB/day) eventually.
- CentOS computer splunkserver.home.loc is running Splunk Enterprise (where logs are kept/analyzed)
- CentOS computer test100.home.loc is running Splunk Forwarder (aka Universal Forwarder) and it will forward its logs to splunk.home.loc.
Get rpm installers
- Download Splunk Enterprise from www.splunk.com/download. Download splunk-6.1.1-207789-linux-2.6-x86_64.rpm
- Download Splunk Forwarder from www.splunk.com/download/universalforwarder. Download splunkforwarder-6.1.1-207789-linux-2.6-x86_64.rpm
Splunk Enterprise: Install on splunkserver.home.loc
- Install rpm.
- Do not change the default install directory. If you do, you will run into issues with upgrades later on.
- Note that user-account/group 'splunk' was created by the splunk installer. And you want to run Splunk Enterprise as user 'splunk', not root.
- Switch from root to user 'splunk' by running:
- As user 'splunk', start Splunk Enterprise for the first time.
- you will see something like this scroll by quickly.
- Next configure the Splunk Enterprise (aka splunk) to start automatically upon reboot.
[[email protected] ~]# rpm -ivh splunk-6.1.1-207789-linux-2.6-x86_64.rpm
[[email protected] ~]# sudo su splunk [[email protected] ~]$ cd
[[email protected] ~]$ /opt/splunk/bin/splunk start --accept-license
[[email protected] ~]$ This appears to be your first time running this version of Splunk. ... The Splunk web interface is at http://splunkserver:8000
Splunk Enterprise: Set service to start automatically upon reboot
- As root, set Splunk Enterprise to auto start upon reboot using user 'splunk'.
- First switch from user 'splunk' back to 'root'. Next as root, run following command
- Now you can use 'service' and 'chkconfig' command to view status, stop/start/restart 'splunk' service.
[[email protected] ~]$ exit exit [[email protected] ~]# /opt/splunk/bin/splunk enable boot-start -user splunk Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.
[[email protected] ~]# service splunk status Splunk status: splunkd is running (PID: 1816). splunk helpers are running (PIDs: 1817 1849). splunkweb is running (PID: 1871). [[email protected] ~]# service splunk restart ... [[email protected] ~]# chkconfig --list splunk splunk 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Splunk Enterprise: Initial login into web interface
- Go to http://splunkserver:8000
- The webpage conveniently shows the default login/password (admin/changeme) you can use to log in for the first time.
- Log in and you will be immediately asked to change the default password.
- Change the password of 'admin'.
- Under Settings | Licensing, the server is currently using license from the Trial license group. By clicking on 'Change license group', you can convert to 'Free license'. I will do that eventually but for now I will use 'Enterprise Trial license'.
Splunk Enterprise: Configure to accept incoming logs from other CentOS computers
- In Splunk Enterprise web interface http://splunkserver:8000, configure the Splunk Enterprise to accept incoming logs.
- Go to Settings | Forwarding and receiving. I'm picking 'Forwarding and receiving' because I will be using splunkforwarder to forward logs from all my Linux computers.
- Click on 'Configure receiving'.
- Click on New.
- Type in a port # to use, in this example 9997.
- Click on Save.
- Now splunkserver is ready to receive logs forwarded by a splunkforwarder.
Splunkforwarder: Install on test100.home.loc
- Install splunkforwarder.
- You can run splunkforwarder service (called splunk) as any user, however this requires giving read access to that user all the logs. So I simply elected to run it as root. As 'root', start splunkforwarder service for first time and you will see following.
- Important: note the login info (admin/changeme) used here is NOT what's on the splunkserver. Splunkforwarder has its own authorized user list, with default administrator account, admin (with default password changeme). Don't make the mistake of trying to use admin and the password that was changed on the http://splunkserver:8000 earlier.
- On test100, specify where the logs should be forwarded to. Note the hostname:9997 in the command. Note 9997 is the port you specified in Splunk server earlier.
- Next change the password of 'admin' in splunkforwarder on test100.home.loc to something else.
- Tell splunkforwarder on test100 to monitor directory /var/log/.
- Restart splunkforwarder service. The service name of splunkforwarder is also 'splunk', same as Splunk Enterprise.
- You can use 'service' and 'chkconfig' to start or monitor 'splunk' service. You will see that 'splunk' is set to auto start upon system reboot.
[[email protected] ~]# rpm -ivh splunkforwarder-6.1.1-207789-linux-2.6-x86_64.rpm
[[email protected] ~]# /opt/splunkforwarder/bin/splunk start --accept-license This appears to be your first time running this version of Splunk. Splunk> Like an F-18, bro. Checking prerequisites... Checking mgmt port : open Creating: /opt/splunkforwarder/var/lib/splunk ... Starting splunk server daemon (splunkd)... Declared role=universal_forwarder. Done
[[email protected] ~]# /opt/splunkforwarder/bin/splunk add forward-server splunkserver.home.loc:9997 -auth admin:changeme
[[email protected] ~]# /opt/splunkforwarder/bin/splunk edit user admin -password YOUR_NEW_PASSWORD -auth admin:changeme
[[email protected] ~]# /opt/splunkforwarder/bin/splunk add monitor /var/log/
[[email protected] ~]# service splunk restart
[[email protected] ~]# service splunk status Splunk status: splunkd is running (PID: 1653). splunk helpers are running (PIDs: 1654). [[email protected] ~]# chkconfig --list splunk splunk 0:off 1:off 2:on 3:on 4:on 5:on 6:off [[email protected] ~]# service splunk restart Restarting Splunk... Stopping splunkd... ...
Splunk Enterprise: View received logs
- Go to http://splunkserver:8000
- Under Apps, click on Search.
- You should see events under 'What to Search'.
- Click on 'Data Summary'.
- Click on the name of the host, test100.
- You can see all the logs that have been forwarded from test100 and received by splunkserver.
- In search box, you will see 'host=test100'.
Splunk Enterprise: Filter out logs
- In the logs, hover cursor over keywords such as '/usr/sbin/crond' to highlight & click.
- In search box, you will see: host=test100 "/usr/sbin/crond".
- Only the logs that match the 2 conditions are displayed.
Splunk Enterprise: View logs in real time
- Arrange Terminal window connected to test100 and the web browser (http://splunkserver:8000) side by side.
- Clear the search box and only leave 'host=test100'.
- Click on 'All time ' button next to the search box and you should see a drop down menu.
- Choose 'All time (real-time)'.
- Restart a service on test100, such as 'service httpd restart'.
- In the web browser (http://splunkserver:8000) you should see logs that were forwarded by test100 immediately.