Vinish JustinVJ
← All writing

Splunk Configuration Precedence: The Complete Cheat Sheet

September 29, 202611 min read2,480 words

  • splunk
  • props.conf
  • transforms.conf
  • configuration
  • cheat-sheet

TL;DR

  • Index-time (global context): system/local → apps/*/local → apps/*/default → system/default. Apps tie-break in ASCII order (0-9 → A-Z → a-z); first wins.
  • Search-time (app/user context): user → current app → other apps (exported only, reverse-lexicographic) → system.
  • props.conf stanzas: [source::] → [host::] → [sourcetype] → [default].
  • Index-time pipeline: line breaking → timestamp → SEDCMD → TRANSFORMS → RULESET → index.
  • Search-time: EXTRACT → REPORT → KV_MODE → FIELDALIAS → EVAL → LOOKUP → eventtypes → tags.
  • Parsing happens once, on the first heavy forwarder or indexer. Search-time settings belong on the search head.
  • Verify everything with splunk btool <conf> list --debug.

Why precedence matters

Splunk merges every copy of a .conf file into one runtime view, and when two copies set the same attribute, precedence decides which value wins. Most “my props.conf isn’t working” tickets are precedence problems, not syntax problems.

There are four layers of “order” to keep in your head:

  1. Directory precedence — which copy of a file wins (system/local vs app/local, etc.).
  2. Stanza precedence — which stanza inside a file wins ([source::] vs [host::] vs [sourcetype]).
  3. Processing order — the order Splunk applies settings at index time and search time (line breaking before TRANSFORMS, EXTRACT before FIELDALIAS, and so on).
  4. Location — which Splunk tier actually reads the setting (UF, HF/indexer, search head).

Directory precedence itself runs in one of two contexts:

  • Global context — index-time and system-wide settings (inputs, outputs, index-time props/transforms, indexes, server).
  • App/user context — search-time settings evaluated for a specific user in a specific app (search-time props, savedsearches, eventtypes, tags, macros).

When in doubt, ask Splunk directly. btool shows the merged result and the file each line came from:

$SPLUNK_HOME/bin/splunk btool props list <stanza> --debug
$SPLUNK_HOME/bin/splunk btool transforms list <stanza> --debug
$SPLUNK_HOME/bin/splunk btool check

btool check validates every stanza and setting name against the .spec files and reports typos or unknown keys. It checks names, not values, and Splunk also runs it automatically at startup.

1. Global context (index-time) precedence

In the global context, system/local wins over everything, and system/default loses to everything.

$SPLUNK_HOME/etc/system/local          <-- highest
$SPLUNK_HOME/etc/apps/<app>/local       (all apps, A → Z)
$SPLUNK_HOME/etc/apps/<app>/default     (all apps, A → Z)
$SPLUNK_HOME/etc/system/default        <-- lowest (never edit)

Tie-breaker between apps: lexicographic (ASCII) order of the app folder name. The app that sorts first wins.

  • ASCII order is: digits 0-9 → uppercase A-Z → lowercase a-z.
  • So 000_org_indexes beats A_org_props, which beats Splunk_TA_windows, which beats org_all_props.
  • This is why admins prefix override apps with 000_, 100_ or an uppercase letter.

Order of evaluation across apps: all apps’ local directories are checked before any app’s default directory. A value in zzz_app/local beats a value in aaa_app/default.

Files that live in the global context include inputs.conf, outputs.conf, indexes.conf, server.conf, limits.conf, authentication.conf, and the index-time parts of props.conf and transforms.conf.

Never edit system/default or an app’s default directory. Upgrades overwrite them.

2. Indexer cluster peers and search head clusters

On an indexer cluster peer, apps pushed by the cluster manager (peer-apps, formerly slave-apps) get an extra slot at the very top.

$SPLUNK_HOME/etc/peer-apps/<app>/local     <-- highest
$SPLUNK_HOME/etc/system/local
$SPLUNK_HOME/etc/apps/<app>/local
$SPLUNK_HOME/etc/peer-apps/<app>/default
$SPLUNK_HOME/etc/apps/<app>/default
$SPLUNK_HOME/etc/system/default            <-- lowest

Practical rules for clusters:

  • Manage peer config only from the cluster manager: etc/manager-apps (formerly master-apps) → bundle push → etc/peer-apps on every peer.
  • A setting in a peer’s system/local can still lose to peer-apps/<app>/local, so don’t “hotfix” a single peer there.
  • Search head cluster: the deployer pushes from etc/shcluster/apps into each member’s etc/apps. By default the deployer merges the app’s local into default on push, so anything a user saves in the UI (member local) wins over deployer content.
  • Deployment server: apps land in the client’s etc/apps and follow normal global-context rules above.

3. App/user context (search-time) precedence

At search time the user’s private copy wins, then the app you are currently in, then everything else.

$SPLUNK_HOME/etc/users/<user>/<current_app>/local   <-- highest
$SPLUNK_HOME/etc/apps/<current_app>/local
$SPLUNK_HOME/etc/apps/<current_app>/default
$SPLUNK_HOME/etc/apps/<other_app>/local        (exported settings only)
$SPLUNK_HOME/etc/apps/<other_app>/default      (exported settings only)
$SPLUNK_HOME/etc/system/local
$SPLUNK_HOME/etc/system/default                 <-- lowest

Three things that trip people up:

  1. The current app beats every other app regardless of its name.
  2. Other apps count only if their objects are exported (export = system in the app’s metadata/*.meta). A non-exported sourcetype extraction in TA_foo is invisible when you search from the Search app.
  3. Between other apps, the order is reverse-lexicographic — the opposite of the global context. An app named B beats an app named A. Don’t rely on this; give conflicting stanzas one owner.

Files in the app/user context include savedsearches.conf, macros.conf, eventtypes.conf, tags.conf, datamodels.conf, ui-prefs.conf, and the search-time parts of props.conf and transforms.conf.

Check from a specific user and app’s point of view:

splunk btool props list <sourcetype> --debug --app=<app> --user=<user>

4. Stanza precedence inside props.conf

After the files are merged, Splunk applies every stanza that matches an event and resolves only the attributes that conflict. The most specific stanza type wins:

[source::<source>]     <-- highest
[host::<host>]
[<sourcetype>]
[default]              <-- lowest (applies to everything)

Within the same stanza type (e.g. two [source::] stanzas that both match):

  1. A literal match beats a wildcard/regex match ([source::/var/log/app.log] beats [source::/var/log/*.log]).
  2. Between two pattern matches, the stanza name that sorts first in ASCII order wins.
  3. Override that ordering with the priority attribute — higher number wins:
[source::.../app/*.log]
priority = 10
TRUNCATE = 20000

[source::.../app/debug*.log]
priority = 20
TRUNCATE = 0

priority only ranks stanzas of the same type. It cannot make a [sourcetype] stanza beat a [source::] stanza.

Other stanza rules:

  • Attributes merge across stanzas. If [source::x] sets only TRUNCATE and [mysourcetype] sets TIME_FORMAT, both apply.
  • A specific stanza overrides the [default] stanza and any global settings at the top of the file.
  • The same stanza name in two files is merged attribute by attribute, and each conflict is settled by directory precedence (sections 1–3).
  • In transforms.conf there’s no stanza precedence: a transform runs only when a TRANSFORMS-/REPORT- line in props.conf calls it by name.

5. Index-time processing order

Index-time settings run in a fixed pipeline order, and you can’t reorder it: line breaking → timestamp → SEDCMD → TRANSFORMS → RULESET → index.

# Pipeline What happens Key props.conf settings
1 Input Source, host, sourcetype, index assigned inputs.conf, [source::] sourcetype=
2 Parsing Character encoding, raw line breaking, truncation CHARSET, LINE_BREAKER, TRUNCATE
3 Merging (aggregator) Line merging, timestamp extraction SHOULD_LINEMERGE, BREAK_ONLY_BEFORE*, MUST_BREAK_AFTER, MAX_EVENTS, TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, TZ, DATETIME_CONFIG
4a Typing Regex replacement on _raw SEDCMD-<class>
4b Typing Index-time transforms: masking, routing, filtering, indexed fields, INGEST_EVAL TRANSFORMS-<class>
4c Typing Ingest actions rulesets RULESET-<class>
5 Indexing Output routing, write to disk _TCP_ROUTING, _SYSLOG_ROUTING, indexes.conf

SEDCMD always runs before TRANSFORMS. If a TRANSFORMS regex looks for text that a SEDCMD already removed or masked, it will never match. You can’t change this order; match on the post-SEDCMD text, or do the edit inside a transform instead.

TRANSFORMS always runs before RULESET. When both apply to the same sourcetype, rulesets see the already-transformed event.

Order among classes

  • Multiple TRANSFORMS-<class> lines run in ASCII order of the class name, e.g. TRANSFORMS-01_route before TRANSFORMS-02_mask.
  • Inside one class, the comma-separated list runs left to right. For queue routing the last matching transform wins:
# props.conf
[my:sourcetype]
TRANSFORMS-filter = setnull, keep_errors

# transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep_errors]
REGEX = ERROR
DEST_KEY = queue
FORMAT = indexQueue

Everything is dropped first, then events containing ERROR are put back. Reverse the order and you keep nothing.

  • Multiple SEDCMD-<class> lines also run in ASCII order of the class name, each on the output of the previous one.
  • Rules inside one ingest actions ruleset run top to bottom, and a sourcetype can have only one ruleset.

The sourcetype-rewrite trap

If a TRANSFORMS rewrites the sourcetype (DEST_KEY = MetaData:Sourcetype), the new sourcetype’s index-time settings do not run — parsing is already finished. Only its search-time settings apply. To process the event again under the new sourcetype, use CLONE_SOURCETYPE.

6. Search-time operation order

Search-time knowledge objects run in a fixed sequence, and each step can only use fields created by the steps before it.

# Operation props.conf / file Can reference fields from
1 Inline field extraction EXTRACT-<class> _raw and indexed fields only
2 Transform-based extraction REPORT-<class> → transforms.conf Step 1
3 Automatic key/value extraction KV_MODE (auto, json, xml, none) Steps 1–2
4 Field aliases FIELDALIAS-<class> Steps 1–3
5 Calculated fields EVAL-<field> Steps 1–4
6 Lookups LOOKUP-<class> Steps 1–5
7 Event types eventtypes.conf Steps 1–6
8 Tags tags.conf Steps 1–7 (incl. eventtypes)

Rules that follow from this order:

  • An EXTRACT can’t use a field produced by REPORT or KV_MODE. Use the in <field> syntax only on fields that already exist.
  • A FIELDALIAS can’t alias a field created by EVAL or a lookup.
  • An EVAL- can use aliases but not other EVAL- fields. Calculated fields are evaluated independently, so chain the logic in one expression.
  • A lookup can use calculated fields and aliases as input keys.
  • Event types can search on lookup output, and tags can be applied to event types.

Order within one step

  • EXTRACT-, REPORT-, FIELDALIAS- and LOOKUP- classes run in ASCII order of the class name.
  • Inside one REPORT- class, transforms run left to right.
  • FIELDALIAS-x = src AS dest overwrites an existing dest; src ASNEW dest leaves an existing dest alone.
  • Lookups use OUTPUT (overwrite existing fields) or OUTPUTNEW (fill only empty fields).

7. Where each setting actually runs

A perfect config on the wrong tier does nothing. Parsing happens once, on the first full Splunk instance (heavy forwarder or indexer) the data reaches.

Setting Universal forwarder Heavy forwarder / indexer Search head
inputs.conf, outputs.conf Yes Yes Only for its own logs
EVENT_BREAKER_ENABLE, EVENT_BREAKER Yes No No
INDEXED_EXTRACTIONS (CSV, JSON, W3C…) Yes Yes (if not already parsed) Also needed here for field display
CHARSET, LINE_BREAKER, SHOULD_LINEMERGE, TRUNCATE No Yes No
TIME_PREFIX, TIME_FORMAT, TZ, MAX_TIMESTAMP_LOOKAHEAD No Yes No
SEDCMD-, TRANSFORMS-, INGEST_EVAL, RULESET- No Yes No
EXTRACT-, REPORT-, KV_MODE, FIELDALIAS-, EVAL-, LOOKUP- No No Yes
eventtypes, tags, macros, savedsearches No No Yes

Consequences worth remembering:

  • Data from a heavy forwarder arrives already parsed (cooked). The indexer skips line breaking, timestamping, SEDCMD and TRANSFORMS for it, so those settings must live on the HF.
  • Data sent from a UF with INDEXED_EXTRACTIONS is also parsed at the UF; index-time props on the indexer won’t reapply.
  • Search-time settings on an indexer only matter if the search head distributes them in the knowledge bundle — put them on the search head.
  • Safe practice: deploy the same TA everywhere and let each tier use the part it reads.

8. Other precedence rules worth knowing

inputs.conf

  • blacklist beats whitelist. A file matching both is skipped.
  • Put the most specific settings on the most specific stanza. A [default] stanza in any app’s inputs.conf merges into every input on that instance, so use it sparingly.
  • Avoid two [monitor://] stanzas that cover the same file — the file is read once and you can’t predict which stanza’s index/sourcetype it gets.

Routing (outputs)

  1. outputs.conf defaultGroup — the fallback.
  2. _TCP_ROUTING on an input in inputs.conf overrides the default group.
  3. A TRANSFORMS- setting DEST_KEY = _TCP_ROUTING (on a HF/indexer) runs later and overrides both.

The same “later wins” logic applies to index (inputs.conf → DEST_KEY = _MetaData:Index) and host/source/sourcetype rewrites.

Permissions (.meta files)

  • metadata/local.meta beats metadata/default.meta.
  • Inside a .meta file, the more specific stanza wins: [props/<stanza>/<attr>] → [props/<stanza>] → [props] → [] (whole app).
  • export = system is what makes an app’s objects visible to other apps at search time (section 3).

Deployment server (serverclass.conf)

  • With the default filterType = whitelist, a client must match a whitelist and not match a blacklist. Blacklist wins.
  • Settings at [serverClass:<name>:app:<app>] override [serverClass:<name>], which overrides [global].

9. One-page quick reference

Topic Order (highest / first → lowest / last)
Global context (index-time) system/local → apps/*/local → apps/*/default → system/default
App tie-break, global ASCII: 0-9 → A-Z → a-z (first wins)
Indexer cluster peer peer-apps/local → system/local → apps/local → peer-apps/default → apps/default → system/default
App/user context (search-time) users/<user>/<app>/local → current app local → current app default → other apps local → other apps default → system/local → system/default
App tie-break, app/user Reverse lexicographic (B beats A); exported objects only
props.conf stanza type [source::] → [host::] → [sourcetype] → [default]
Same stanza type literal → pattern; higher priority wins; else ASCII
Index-time pipeline Input → Parsing (LINE_BREAKER) → Merging (LINEMERGE, timestamp) → Typing (SEDCMD → TRANSFORMS → RULESET) → Indexing
Multiple TRANSFORMS-/SEDCMD- classes ASCII order of class name; within a class, left to right
Search-time EXTRACT → REPORT → KV_MODE → FIELDALIAS → EVAL → LOOKUP → eventtypes → tags
inputs / serverclass filters blacklist beats whitelist
Output routing defaultGroup → inputs _TCP_ROUTING → TRANSFORMS _TCP_ROUTING (last wins)
.meta permissions local.meta → default.meta; [type/stanza/attr] → [type/stanza] → [type] → []

btool commands

# Merged view + source file of every line
splunk btool props list <sourcetype> --debug
splunk btool transforms list <stanza> --debug
splunk btool inputs list --debug | grep <path>

# Search-time view for one user in one app
splunk btool props list <sourcetype> --debug --app=search --user=admin

# Syntax / typo check across all .conf files
splunk btool check

Sources