Splunk daily health dashboard: implementation guide

Audience: Splunk core administrators Time required: about 60 minutes Splunk version: 9.x (see step 6 for 8.x) Source file: splunk_daily_health_dashboard.json (52 panels, 8 sections)

---

Before you start

You need:

• The source file, splunk_daily_health_dashboard.json. [https://vinishjustin.dev/downloads/splunk_daily_health_dashboard.json]
• A role with search access to _internal, _introspection, and _audit. Without them, several panels stay empty.
• Access to run searches on the search head, and ideally on the cluster manager, the license manager, and a search head cluster member.
• A non-production app for the first install. Move it once it works.

Step 1: Confirm access

Run this search. It should return a count without an error.

index=_internal earliest=-15m | stats count by sourcetype

Check the REST access that several panels use. If this returns an error, ask your Splunk admin to grant the rest_properties_get capability, or run the dashboard from an account that has it.

| rest /services/server/info splunk_server=local | table splunk_server version

Step 2: Create the dashboard in Dashboard Studio.

1. Go to Dashboards, then select Create New Dashboard.
2. Enter a title, such as Splunk admin: daily health check.
3. Choose Dashboard Studio and select the Absolute layout. The source uses absolute positioning, so the layout must match.
4. Select Create.
5. Open the source editor (the </> button), delete the default content, and paste the full JSON from the file.
6. Select Back, check that the panels render, then select Save.

Check after saving: the page shows eight sections and a time range picker at the top. A blank page or a JSON error means the paste was incomplete. Copy the file again and retry.

Step 3: Set the inputs

Input	Default	What it controls
Time range	Last 24 hours, snapped to the hour	Window for the summary tiles and trend charts
Silent host threshold	4 hours	How long a host can go without data before it is flagged

Pick a threshold that matches how each source reports. A source that sends every five minutes can be flagged after one hour. A daily batch source needs a much longer threshold, or it shows as silent every afternoon.

Step 4: Run it where it belongs.

Section	Run from	Why
1 to 5: platform, ingestion, license, search, infrastructure	Search head	Searches the indexers and reads scheduler and license data
License tiles	License manager, or a search head that receives its internal logs	License usage is logged where the license manager runs
6: clustering	Cluster manager	Peer, fixup, and replication endpoints live on the manager
8: search head cluster and bundles	Any search head cluster member	Each member reports its own view of peers and bundle replication

On a standalone search head, sections 6 and 8 show empty or zero values. That is expected. Remove those panels rather than leaving misleading zeros on the page.

Step 5: Verify each search

Copy the search behind each tile into a new search and run it. These five cover the most common failures.

Indexing rate for today:

index=_internal source=*license_usage.log* type=Usage earliest=@d
| stats sum(b) as bytes
| eval GB=round(bytes/1073741824,2)

Blocked queues, last 15 minutes:

index=_internal source=*metrics.log* group=queue blocked=true earliest=-15m
| stats count as blocked_events

Silent hosts over the last 7 days:

| tstats latest(_time) as last_seen where index=* by host
| eval hours_silent=round((now()-last_seen)/3600,1)
| where hours_silent>4

Scheduler skips:

index=_internal sourcetype=scheduler status=skipped earliest=-24h
| stats count by savedsearch_name, reason

splunkd errors:

index=_internal sourcetype=splunkd log_level=ERROR earliest=-24h
| stats count by component

If a search returns nothing, check for one of three causes: the role lacks index access, the index name differs in your environment, or the window is too short for the data to exist yet.

Step 6: Adjust for your Splunk version

The cluster searches use the /cluster/manager/ endpoints, which are current in Splunk 9.x. On Splunk 8.x, replace manager with master in the clustering searches. Search the source for cluster/manager and change each occurrence.

Confirm the account can reach those endpoints, on the cluster manager:

| rest splunk_server=local /services/cluster/manager/info | table replication_factor search_factor

Step 7: Set thresholds from your baseline

The color ranges in the source are starting points. Set them after a week of data.

1. Open the dashboard for the last seven days and note the normal range of each tile.
2. Set green to cover normal operation.
3. Set amber where an operator should look, and red where someone must act.
4. Edit the colorConfig values in the source, then save.

The tiles most worth tuning are license used today, blocked queue events, and splunkd errors. A busy deployment may run a steady number of errors that is still healthy, so a fixed starting value can cause false alarms.

Step 8: Share and set permissions

• Set sharing to App if the whole team should see the dashboard, or Private if it is only for you.
• Give read access to the admin role and to any on-call role that reviews it.
• Keep the dashboard out of the default search app if it will be used daily.

Each viewer sees results based on their own permissions. A user without access to _audit sees the failed login panel empty, which is correct.

Step 9: Run the daily routine

Morning checklist, about 15 minutes:

• Platform tiles: all instances reachable, no blocked queues, no critical disk.
• License: usage within quota, and no new warnings in the license messages table.
• Scheduler: skipped searches are explained, or each has a ticket.
• Clustering: replication and search factors met, no peers down.
• Bundles: no peer failing replication since the last check.
• Errors: any new error component since yesterday has been reviewed.

Record anything red and the action you took. The dashboard shows what is happening, and the decision log helps when the same problem returns.

Troubleshooting

Symptom	Likely cause	Fix
Blank page after pasting	Incomplete or altered JSON	Copy the whole file again and check the end of the paste
Panels show “No results”	Index access or wrong time range	Run the search in Search with the same window; check index permissions
Cluster panels empty	Dashboard runs on a standalone search head	Run from the cluster manager, or remove section 6
Bundle panels show zero	Not on an SHC member, or no distributed search	Run from an SHC member, or remove section 8
Silent hosts flood the table	Threshold too short for the sources	Raise the silent host threshold input
Latency table is slow	Searching all indexes over 15 minutes	Restrict the search to the indexes that matter
Clustering values look wrong on 8.x	Endpoint name changed	Use master in place of manager

---

Once the dashboard runs cleanly for a week, consider a weekly companion covering index growth, bucket health, and knowledge object hygiene. Use the same method: one JSON source, one search per tile, and a checklist for review.