Vinish JustinVJ
← All writing

How to Regenerate GUID in Splunk Universal Forwarder via Agent Management

October 5, 20267 min read1,480 words

  • splunk
  • universal-forwarder
  • agent-management

In this article, we are going to see how to regenerate the GUID in a Splunk Universal Forwarder (UF). This is a very common requirement in real projects, especially when servers are built from a cloned VM, a template or a golden image. So let’s understand the problem first, and then we will see the solution step by step.

What is the GUID in Splunk?

Every Splunk instance has one unique identifier called the GUID. It is stored in this file:

$SPLUNK_HOME/etc/instance.cfg

Splunk creates this GUID automatically when it starts for the first time. The idea is simple: one machine, one GUID.

What is the problem with duplicate GUIDs?

Now think about this. Your team takes a VM that already has the forwarder installed and started, and then clones it 50 times. All 50 servers now have the same GUID. Splunk thinks they are all the same machine.

Because of this, you may face issues like:

  • Deployment Server confusion: it tracks clients by GUID, so many hosts show up as a single client, or the names keep changing.
  • Wrong monitoring data: the Monitoring Console shows a wrong count of forwarders or wrong host names.
  • Difficult troubleshooting: searching by guid in the _internal index no longer points to one machine.

So the GUID must be unique for each server. Let’s see how to fix it.

Method 1: Use splunk clone-prep-clear-config (best method for new images)

If you are still preparing the template or golden image, this is the recommended way. Splunk gives us a built-in CLI command for this:

$SPLUNK_HOME/bin/splunk stop
$SPLUNK_HOME/bin/splunk clone-prep-clear-config

This command clears the instance-specific settings, such as the GUID in instance.cfg, and identity settings like serverName and the default host value. (Exact list can change between versions, so please check the Splunk documentation of your version once.)

Important points to remember:

  1. Run this command only when Splunk is stopped.
  2. After running it, do not start Splunk again on the template machine. If you start it, a new GUID is generated again, and the whole purpose is lost.
  3. Now capture the image or take the clone. Every new server will generate its own fresh GUID on its first start.

So if you are building a new image, always do this step. Prevention is better than cure!

Method 2: Reset the GUID on forwarders that are already running

Download the App : https://vinishjustin.dev/downloads/reset_guid_uf.zip

Now, what if the servers are already cloned and the forwarders are already running? In that case we have to reset the GUID manually. The basic process is very simple:

  1. Delete the file $SPLUNK_HOME/etc/instance.cfg
  2. Restart the forwarder. Splunk will create a new GUID at startup.

But doing this manually on hundreds of servers is not practical. Also, there is one small trouble: the restart. If a script launched by Splunk tries to restart Splunk, the script itself may get killed when Splunk stops.

So the best approach is to create a small one-shot app. This app will:

  1. Wait for a few seconds after startup
  2. Take a backup of instance.cfg and remove it
  3. Disable itself, so that it never runs again
  4. Restart the forwarder

You deploy the app once, it does its job one time, and then it switches itself off.

App folder structure

reset_guid_uf/
├── default/
│   ├── app.conf
│   └── inputs.conf
└── bin/
    └── (scripts for your platform)

inputs.conf

[script://$SPLUNK_HOME/etc/apps/reset_guid_uf/bin/reset_guid.sh]
disabled = 0
interval = -1
index = _internal
sourcetype = reset_guid_uf

Here interval = -1 is the key setting. It means “run the script only once at startup, and don’t repeat”.

Safety points to keep in mind

  • Take a backup of instance.cfg before deleting. If anything goes wrong, you can roll back.
  • Create a marker file on first run. If the marker already exists, the script exits immediately. This protects you from running twice.
  • Disable the app before the restart, not after. Write local/app.conf with [install] and state = disabled.
  • Run the worker detached from splunkd, so it does not get killed when Splunk stops.
  • Write a log file under $SPLUNK_HOME/var/log/splunk/, so you can verify each step later.

Linux servers

Here is a simple shell worker script that does the core steps:

#!/bin/sh
SPLUNK_HOME="${SPLUNK_HOME:-/opt/splunkforwarder}"
APP_LOCAL="$SPLUNK_HOME/etc/apps/reset_guid_uf/local"
LOG="$SPLUNK_HOME/var/log/splunk/reset_guid_uf.log"
ts() { date '+%Y-%m-%d %H:%M:%S'; }

{
  echo "$(ts) starting GUID reset"
  sleep 15

  cp -p "$SPLUNK_HOME/etc/instance.cfg" "$SPLUNK_HOME/etc/instance.cfg.bak.$(date +%Y%m%d%H%M%S)"
  rm -f "$SPLUNK_HOME/etc/instance.cfg"

  mkdir -p "$APP_LOCAL"
  printf '[install]\nstate = disabled\n' > "$APP_LOCAL/app.conf"

  "$SPLUNK_HOME/bin/splunk" restart
} >> "$LOG" 2>&1

Start this script in the background using nohup (and setsid if available), so it stays alive when splunkd stops.

What if Splunk is running as a systemd service?

This is a very common situation. When the UF runs under systemd, the splunk user normally does not have permission to run systemctl restart. To give it permission you need a polkit rule or a sudoers entry, which means changing the system configuration. Many teams don’t want this.

No problem, there is another way. The splunk user owns the splunkd process, so the script can simply stop it:

PID=$(head -1 "$SPLUNK_HOME/var/run/splunk/splunkd.pid")
kill -TERM "$PID"

After that, systemd will start the service again by itself using its restart policy. But this works only if the unit file has Restart=always, which is the usual Splunk default. Please confirm first:

systemctl show SplunkForwarder -p Restart

If the output is on-failure or no, then a clean SIGTERM will not bring the service back, and the forwarder will stay down. In that case, go with the polkit or sudoers option.

AIX servers

AIX does not use systemd, and the splunk user can restart its own process directly. So the same shell approach works, but keep these small differences in mind:

  • Use plain /bin/sh (ksh) and avoid bash-only features.
  • The AIX date command does not support %F and %T. Use date '+%Y-%m-%d %H:%M:%S' instead.
  • There is no setsid binary on AIX. Use perl’s POSIX::setsid() if perl is installed, otherwise use nohup.
  • AIX tar does not have the z option. Extract like this: gunzip -c app.tgz | tar xf -

Windows servers

On Windows, the same logic works with PowerShell and the service manager. The scripted input calls a .bat file, which calls a PowerShell launcher. The launcher starts the worker through WMI, so the worker is not a child of the Splunk service:

$cmd = "powershell.exe -NoProfile -ExecutionPolicy Bypass -File `"$Worker`""
Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments @{ CommandLine = $cmd }

The worker takes a backup, removes etc\instance.cfg, writes the disabled local\app.conf, and then restarts the service:

Restart-Service -Name SplunkForwarder -Force

The service account of the forwarder must have permission to restart its own service. The default LocalSystem account can do this. If you use a restricted account, the GUID will already be removed and the app disabled, so just restart the service manually to finish the job.

How to deploy the app

  1. Copy the app into $SPLUNK_HOME/etc/apps/.
  2. On Linux and AIX, make sure the scripts are executable and owned by the Splunk user.
  3. Restart the forwarder once, so the app gets loaded.
  4. After about 15 seconds the script runs, the GUID is reset, and the forwarder restarts.

If you are pushing the app from the Deployment Server, remember that it will be redeployed again and again. So remove it from the server class after it has run. Otherwise, only the marker file protects you from a repeat run.

How to verify

Please check these points after the activity:

  • instance.cfg should have a new guid value.
  • The log file reset_guid_uf.log should show all the steps completed.
  • local/app.conf in the app should show state = disabled.
  • On the Deployment Server, the client should now appear as its own separate entry.

Once everything is confirmed, you can delete the app completely.

Best practices

  • Prevent, don’t fix. Run splunk clone-prep-clear-config before creating any image, and don’t start Splunk on that template afterwards.
  • Test on one server first. Permissions and process handling can be different in every environment.
  • Keep the backup file. The instance.cfg.bak.* file costs nothing, and rollback becomes easy.
  • Never keep the app enabled. If the reset runs on every start, the server will get a new identity every time.

Conclusion

Duplicate GUIDs look like a small thing, but they can create big confusion in your Splunk environment. Use clone-prep-clear-config while building new images, and use the self-disabling one-shot app for forwarders that are already running in the field.

I hope this article helped you. If you have any doubts or questions, please write them in the comments section below, and don’t forget to share this with your team members. Thank you for reading!