Splunk Sidecars Explained in Simple Terms

A Few weeks ago, one of my colleagues pinged me with a list of issues he noticed on the Splunk servers:

Our endpoint security tool raised alerts for new processes like nascent and spotlightcollector on the Splunk hosts.
After running splunk stop, some processes like ipc_broker and opamp-svc were still showing in the ps -ef output.
A few unexpected ports were showing as occupied, and the network team asked him what they were being used for.
He had moved the Spotlight log folder to a different path, and after that the diag file was not collecting the sidecar metrics.

Actually, all four issues have the same answer. These processes are called sidecars, and from the newer Splunk versions they are a normal part of the setup. Many admins are getting confused about this, so I thought of writing a simple blog on it.

In this blog we will see what sidecars are, why Splunk is using them, how they are managed, and what you should take care of as an admin.

What is a sidecar?

You must have seen the old Bajaj or Royal Enfield bikes with a sidecar attached. The bike does the main job of running on the road, and the sidecar just goes along with it to carry one extra person or some luggage.

Splunk sidecars work in the same way. The main process is splunkd. Sidecars are separate processes which run along with splunkd, and each one does one particular job. They are long-running processes, meaning they keep running as long as Splunk is running, and they need to be monitored so that they stay healthy.

Sidecars are available in both Splunk Enterprise and Splunk Cloud Platform.

Why is Splunk using sidecars?

If Splunk keeps adding every new feature inside splunkd itself, that one process becomes very heavy and difficult to change. So instead, Splunk is moving features into separate small processes. The benefits are:

New features can be developed and released easily, because each one is in its own process.
Workload gets distributed instead of everything sitting on one process.
Performance of the system improves, since each process concentrates on its own work.

Most of the sidecars available now are supporting the enhanced data management features in on-premises Splunk Enterprise.

How are sidecars managed?

Let us understand this with a simple example. Think of a hotel.

The owner is splunkd.
The owner appoints a manager. In Splunk, this manager is called the supervisor (process name splunksupervisor).
The manager handles all the staff. These are the sidecars.

So the flow is like this:

splunkd starts the supervisor.
The supervisor starts the sidecars, monitors them, and sends metrics about them.
If any sidecar becomes unhealthy or stops, the supervisor restarts it. But this happens only if that sidecar is listed in the manifest.yaml file. This file has the list of all sidecars which come with Splunk Enterprise.

Also note that there are two types of sidecars. Some sidecars provide features directly. Some sidecars are there only to support the other sidecars, for example IPC Broker and Spotlight.

What happens when splunkd stops?

Now comes the interesting part, and many people get confused here.

When splunkd stops, the supervisor and sidecars may still continue running. Mainly this happens in two situations:

**splunkd crashed or was killed.** In this case the child processes are not automatically stopped.
Some sidecar takes too much time to stop after you run splunk stop. Because of this, the supervisor cannot shut down cleanly. So splunkd directly kills the supervisor, and some sidecars may be left running.

No need to worry about this. When splunkd starts again, the supervisor also comes back, it finds those leftover sidecars and restarts them properly.

List of sidecars

Below is the list of sidecars you may see on your servers.

Sidecar	Process name	What it does	Where it runs
Supervisor	`splunksupervisor`	Starts and monitors sidecars, sends metrics	All
SCIM identity	n/a	Deletes users who are removed from your identity provider, using the SCIM standard	Splunk platform
Storage	`postgres`	Manages a PostgreSQL 17.6 instance for enhanced data management	Linux only
Data Orchestration	`cmporchestrator`	Runs SPL2 searches, collects results, manages search metadata and SPL2 module files	Linux only, Splunk Enterprise
Edge Processor Control Plane	`edgeprocessorconfig`	Gives Splunk Enterprise users access to Edge Processor features	Linux only, single search heads
OpAmp	`opamp-svc`	Manages remote agents like Edge Processor or OpenTelemetry Collector using the OpAmp protocol	Linux only, single search head or single deployment server
Agent Management	`agentmanager`	Manages a large number of Splunk agents like Universal Forwarders	Splunk Enterprise
IPC Broker	`ipc_broker`	Handles communication between sidecars, like assigning and discovering ports	Splunk platform
Spotlight	`spotlightcollector`	Collects and saves metrics from the supervisor and sidecars	Splunk platform
Nascent	`nascent`	Makes sure etcd runs with the correct configuration on search head clusters	Search head clusters

Understanding the Nascent sidecar

Out of all these, I found Nascent quite interesting, so let us look at it in a little more detail.

Nascent manages etcd. etcd is a distributed key-value store. Basically it stores configuration data and helps services find each other inside a cluster. To keep the data consistent across all nodes, it uses the Raft consensus algorithm.

Nascent starts etcd on every search head in the search head cluster (SHC). But all search heads are not given the same role:

Maximum 5 search heads become actual etcd cluster nodes. Nascent gives them the flavor **full**. These nodes run the real etcd server, which stores and replicates the data. They also have the etcd_data directory.
All remaining search heads become etcd proxy nodes, with the flavor **proxy**. These nodes run the etcd gateway, which simply forwards requests to the real etcd servers.

Think of it like a bank branch. Only a few staff maintain the main ledger, and the others at the front desk just take your request and pass it to them. Why only 5? Because in consensus systems like Raft, if you keep adding more voting members, everything becomes slower. So it is better to keep a small core team and let the others act as forwarders.

Nascent saves this information in the node_flavor.json file. For example, in a cluster with 7 search heads, it will look something like this:

[
  {
    "name": "etcd",
    "data": {
      "full": ["sh1", "sh6", "sh3", "sh2", "sh4"],
      "proxy": ["sh5", "sh7"]
    }
  }
]

Here 5 search heads are storing the data, and the remaining 2 are just forwarding the requests.

Points to remember as a Splunk admin

Process tree will look bigger. You will see sidecars as child processes of splunkd. This is expected, don’t panic.
Sidecars can occupy network ports. If you are monitoring open ports or have strict firewall rules, keep this in mind.
Security tools may raise alerts. Endpoint security scanners on on-premises servers may flag these new processes. Better to inform your security team in advance so that they don’t raise a ticket for nothing.
Don’t change the Spotlight folder path without thinking. By default, Spotlight saves metrics in spotlight_metrics.ext under $SPLUNK_HOME/var/log/spotlight. If you change this path, the diag file and RapidDiag app may not collect these metrics, and troubleshooting will become difficult.
Many sidecars work only on Linux. Storage, Data Orchestration, Edge Processor Control Plane and OpAmp will not run on other operating systems.
Sidecars running after stop is not always an issue. When splunkd starts again, they will be taken care of.

Conclusion

In short, sidecars are Splunk’s way of adding new features without making splunkd heavy. The supervisor keeps them running, manifest.yaml decides which sidecars are there, and each sidecar does its own job, whether it is running PostgreSQL or maintaining etcd in your search head cluster.

So next time your security team raises an alert for postgres or nascent, or you see ipc_broker still running after a stop, you will know what is happening. It is just Splunk’s sidecar doing its duty.

Hope this blog helped you understand sidecars clearly. If you have any doubts, please drop them in the comments below.

Happy Splunking!