Vinish JustinVJ
← All writing

Splunk Universal Forwarder Troubleshooting

November 5, 20242 min read533 words

Deployment Client Side

  1. Check Service status
    • Windows
      • Services -> SplunkForwarder [Status: Running]
      • Else Start Service
    • Linux
      • /opt/splunkforwarder/bin/splunk status
      • /opt/splunkforwarder/bin/splunk start
      • /opt/splunkforwarder/bin/splunk restart
      • If systemctl configured
        • systemctl status splunkforwader
        • systemctl start splunkforwader
    • AIX
      • /opt/splunkforwarder/bin/splunk status
      • /opt/splunkforwarder/bin/splunk start
      • /opt/splunkforwarder/bin/splunk restart
      • stopsrc -a splunk
      • startsrc -a splunk
  2. Check the Deployment Client configuration [verify targetUri Entry]
    • Windows
      • C:\Program Files\splunkforwarder\etc\system\local\deploymentclient.conf
      • C:\Program Files\splunkforwarder\etc\deploymentclientapp>\local\deploymentclient.conf
    • Linux/AIX
      • /opt/splunkforwarder/splunk btool deploymentclient list –debug
      • /opt/splunkforwarder/etc/system/local/deploymentclient.conf
      • /opt/ splunkforwarder/etc/apps/<deploymentclientapp>/local/ deploymentclient.conf
  3. Check the ownership [If Splunk is running as non-root user]
    • Linux/AIX
      • chown -R splunk:splunk /opt/splunkforwarder
  4. Check mod of the file reading path [If Splunk is running as non-root user]
    • Linux
      • To set acl to directory recursively:
        • setfacl -R -m u:splunk:r /var/log
      • To set acl for individual files:
        • setfacl -m u:splunk:r /var/log/messages ll /var/log/messages
      • To view the ACL
        • getfacl -R /var/log
    • AIX
      • To change a directory’s access:
        • setfacl -m splunk:r /var/log
      • To view the ACL
        • getfacl /var/log
    • chmod 644 /var/log [If no other way 😊]
  5. Network [Ports used in command are default one change as per yours]
    • Connectivity
      • Windows
        • telnet <deploymentserverIp> 8089
        • Test-Connection <deploymentserverIp> -TCPPort 8089 [Powershell]
      • Linux/AIX
        • telnet <deploymentserverIp> 8089
        • nc -nv <deploymentserverIp> 8089
        • nc -s <sourceip> <deploymentserverIp> 8089
      • If logs not received, change <deploymentserverIp> with <indexerIP> and 8089 with 9997
      • If you couldn’t connect check firewall [Local Firewall also]
    • Local Firewall
      • Windows
        • netsh advfirewall show allprofiles state
      • Linux
        • systemctl status firewalld
      • AIX
        • ipfstat
    • DNS [If you’re using hostname instead IP]
      • Windows/Linux/AIX
        • nslookup <deploymentserverhostname>
      • If not resolving IP, check DNS / Host entry
      • Windows
        • ipconfig /displaydns
        • Make Entry in C:\Windows\System32\drivers\etc\hosts
          • 10.10.10.10 xxxxxx.xxxx.xx xxxxxx
      • Linux/AIX
        • cat /etc/resolv.conf
        • edit /etc/hosts
          • 10.10.10.10 xxxxxx.xxxx.xx xxxxxx
    • Note: If you’re using NAT, Use IP address as per that, instead of original IP
  6. Check Connection status
    • Windows
      • netstat -na | find “8089”
      • netstat -na | Select-String “8089” [Powershell]
    • Linux/AIX
      • netstat -an |grep 8089
      • tcpdump -i any host <deploymentserverIp>
  7. Check the logs
    • Windows
      • C:\Program Files\Splunkforwarder\var\logs\splunk\splunkd.log
    • Linux/AIX
      • /opt/splunkforwarder/var/logs/splunkd.log

Deployment SeRVER Side

  1. Check the Connectivity
    • Windows
      • Use Wireshark
    • Linux
      • tcpdump -i any host <deploymentclient IP>
    • If you can see traffic, connectivity is fine.
  2. Check the Forwarder Management
    • In Clients, mention host or IP of the deployment client
    • verify the Machine Type and Phone Home
    • Check the server class part of the particular deployment client
      • Verify the clients
      • Confirm the apps assigned
  3. From Search
    • index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host
    • | rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | rex field=applications “applications\(?.+)\serverclasses” | stats dc(hostname) as hostname by apps
  4. From CLI
    • /opt/splunk/bin/splunk list deploy-clients

Incorporating best practices like using splunk btool, splunk diag, and monitoring internal logs can significantly improve your ability to troubleshoot efficiently.

Happy troubleshooting!