Managing Splunk with Ansible, the Easy Way

It is Friday evening, 7 PM. You are already dreaming of weekend. Then your manager pings: “Hey, just update SplunkForwarders on all 20 production servers before you leave, okay?” Just. That one word has ruined many weekends.

The old way: SSH into server 1, run commands. SSH into server 2, run the same commands. By server 14 you have made one typo, and by server 20 you have missed your train. Ansible fixes exactly this problem. You write the steps once, and Ansible runs them everywhere — same way, every time.

What is Ansible, in simple words?

Ansible is an open-source automation tool. You describe the desired state of your servers in plain YAML, and Ansible connects over SSH and makes it happen. It is agentless — nothing to install on the servers themselves.

One control node, a simple inventory, and every change reaches the right server.

The control node holds Ansible, the inventory and the playbooks, and pushes each task out over SSH; the managed servers only need SSH and Python.

Step 1: Install Ansible on the control node

You install Ansible only on the control node — your laptop or one admin server. The managed servers need nothing extra, just SSH and Python, which almost every Linux box already has. Note: the control node must be Linux or macOS. Windows users, use WSL — no tension.

sudo dnf install -y epel-release
sudo dnf install -y ansible

ansible --version

If you see a version number and a config file path, congratulations — step one is done.

Step 2: Set up passwordless SSH

Generate a key on the control node (skip if you already have one):

ssh-keygen -t ed25519 -C "ansible-control"

Copy it to each server (you need to enter the password):

ssh-copy-id splunkroot@192.168.1.11
ssh-copy-id splunkroot@192.168.1.12
ssh-copy-id splunkroot@192.168.1.21

Now test it — this should log you in without asking any password:

ssh splunkroot@192.168.1.11

Note: Use a dedicated user with sudo rights on every server.

Step 3: Create the inventory file

vi inventory.ini

[webservers]
web1 ansible_host=192.168.1.11
web2 ansible_host=192.168.1.12

[dbservers]
db1 ansible_host=192.168.1.21

[production:children]
webservers
dbservers

[all:vars]
ansible_user=devops
ansible_python_interpreter=/usr/bin/python3

Step 4: Write your ansible.cfg

Instead of typing -i inventory.ini with every command, put your defaults in ansible.cfg inside the project folder. Ansible picks it up automatically when you run commands from that folder.

vi ansible.cfg

[defaults]
inventory = ./inventory.ini
remote_user = devops
host_key_checking = False
forks = 10
retry_files_enabled = False

[privilege_escalation]
become = True
become_method = sudo
become_user = root
become_ask_pass = False

forks = 10 means Ansible works on 10 servers in parallel. Increase it for big fleets.
become = True means tasks run with sudo by default.
host_key_checking = False is fine for a lab. In production, better to keep it on and add host keys properly.

Step 5: Say hello with ad-hoc commands

Ping every server in one shot:

ansible all -m ping

A few more one-liners worth keeping in your pocket:

# Check uptime on web servers
ansible webservers -m command -a "uptime"

# Check free disk space everywhere
ansible all -m shell -a "df -h /"

# Install a package on the database servers
ansible dbservers -m apt -a "name=htop state=present"

# Restart SplunkForwarder on web servers
ansible webservers -m service -a "name=SplunkForwarder state=restarted"

Step 6: Write your Splunk upgrade playbook

Refer:

Splunk Forwarder: splunkforwarder-upgrade.yml
Splunk Enterprise: splunk-upgrade.yml

First do a dry run, then the real thing:

ansible-playbook splunkforwarder-upgrade.yml

ansible-playbook splunkforwarder-upgrade.yml

ansible-playbook -i "192.168.1.11," splunkforwarder-upgrade.yml

Wrapping up

Install Ansible on one machine, set up SSH keys, list your servers in an inventory, and write a playbook. That Friday-evening “update all 20 servers” request is now a two-minute job.

If this post helped you, share it with that one colleague who still SSHs into servers one by one. You know who I’m talking about. Happy automating and Happy Splunking!