Vinish JustinVJ
← All writing

AIX Security Logging to Splunk: Single-Line Audit, sudo and cron Events in /var/log/messages

October 26, 202614 min read2,990 words

  • AIX
  • Splunk
  • Security
  • Audit
  • syslog

Summary

AIX does not log most security activity to syslog by default. Logins appear, but executed commands, process activity, crontab changes, failed logins and su history live in the audit subsystem or in binary files. This guide brings all of it into one file, /var/log/messages, with one line per event, so the Splunk Universal Forwarder (UF) can collect it without any remote syslog configuration.

What you get:

  • Authentication: SSH logins and failures, su, password and account changes
  • Commands executed: every exec by root and admin users (PROC_Execute)
  • Process activity: create, kill, privilege and user ID changes
  • sudo: every allowed and denied command
  • cron: job runs, plus who added or removed crontab entries
  • Binary logs: failedlogin, wtmp, sulog and errpt
  • Kernel privilege messages: with the username resolved in Splunk

Note: AIX does not use auditd, which is the Linux audit daemon. AIX has its own audit subsystem, controlled by the audit command and configured in /etc/security/audit/.

Architecture

sshd / su / login ──┐
sudo ───────────────┤
AIX audit stream ───┤──►  /var/log/messages  ──►  Splunk UF  ──►  Splunk indexer
cron log forwarder ─┤      (single-line events)    (monitor)
binary log forwarder┘

No @host remote syslog entry is used. The UF is the only transport.

Tag / program Facility Producer Covers
sshd, su, login auth syslogd (native) SSH logins, failures, sessions
sudo auth sudo (native) Elevated commands, denials
AIXAUDIT local6 audit2syslog.sh (direct write) Logins, su, commands, processes, files, crontab changes
AIXCRON local5 cron2syslog.sh cron and at job runs and results
AIX_FAILEDLOGIN / AIX_WTMP / AIX_SULOG / AIX_ERRPT auth seclog2syslog.sh Failed logins, sessions, su, system errors
unix (privilege command) kern AIX kernel (native) Privileged commands by UID

Step 0: Prerequisites and backup

  • Root access and an approved change window
  • AIX 7.1, 7.2 or 7.3
  • sudo 1.8 or newer from the AIX Toolbox (check with sudo -V)
  • At least 1 GB free in /var (check with df -g /var)
  • The Splunk UF package for AIX, and the index name from your Splunk team
  • If audit is already used for compliance in bin mode, keep binmode = on in Step 2
B=/root/logcfg_backup_$(date +%Y%m%d)
mkdir -p $B
cp -p /etc/syslog.conf               $B/
cp -p /etc/security/audit/config     $B/
cp -p /etc/security/audit/streamcmds $B/
cp -p /etc/sudoers                   $B/
cp -p /etc/inittab                   $B/
crontab -l    > $B/root.crontab
audit query   > $B/audit_query.txt 2>&1

Step 1: syslogd writes everything to /var/log/messages

Add this single line to /etc/syslog.conf:

*.info;mail.none   /var/log/messages   rotate size 50m files 10
touch /var/log/messages
chmod 640 /var/log/messages
refresh -s syslogd
logger -p auth.info "SYSLOG_TEST auth"
tail -3 /var/log/messages

*.info already covers auth (sshd, su, sudo) and local5 (cron). Do not add separate auth.info or local5.info lines for the same file. syslogd writes once for each matching line, so every event would be duplicated.

Step 2: Audit classes, users and stream mode

Edit /etc/security/audit/config:

start:
        binmode = off
        streammode = on

streammode:
        cmds = /etc/security/audit/streamcmds

classes:
        auth  = USER_Login,USER_Logout,TERM_Logout,USER_SU,PASSWORD_Change,PASSWORD_Flags
        acct  = USER_Create,USER_Remove,USER_Change,GROUP_Create,GROUP_Change,GROUP_Remove
        proc  = PROC_Create,PROC_Execute,PROC_Delete,PROC_Kill,PROC_SetUserIDs,PROC_Privilege
        files = FILE_Unlink,FILE_Rename,FILE_Mode,FILE_Owner,S_PASSWD_WRITE,S_USER_WRITE
        net   = TCPIP_connect,TCPIP_access
        cron  = CRON_JobAdd,CRON_JobRemove,AT_JobAdd,AT_JobRemove

users:
        root    = auth,acct,proc,files,net,cron
        default = auth,acct,proc,cron

Every event name must exist in /etc/security/audit/events, or audit start fails. For example:

grep -w PROC_Execute /etc/security/audit/events

Step 3: Audit stream pipeline, one line per event

/etc/security/audit/streamcmds:

/usr/sbin/auditstream | /usr/sbin/auditpr -h elRtcrpP -v | /usr/local/bin/audit2syslog.sh &

auditpr -v prints each event on two lines: a header and an indented detail line. The script joins them into one line and appends it directly to /var/log/messages in syslog format.

/usr/local/bin/audit2syslog.sh:

#!/usr/bin/ksh
# Joins each auditpr record (header + detail line) into one line
# and appends it directly to /var/log/messages in syslog format.
LOGF=/var/log/messages
HOST=$(/usr/bin/hostname | /usr/bin/cut -d. -f1)

exec /usr/bin/awk -v logf="$LOGF" -v host="$HOST" '
BEGIN { "date +\"%b %e %H:%M:%S\"" | getline last_ts }

function emit(rec,   ts, a) {
  if (rec == "") return
  ts = last_ts
  # Use the event time from the audit record itself
  if (match(rec, /[A-Z][a-z][a-z] [A-Z][a-z][a-z] +[0-9]+ [0-9][0-9]:[0-9][0-9]:[0-9][0-9] [0-9][0-9][0-9][0-9]/)) {
    split(substr(rec, RSTART, RLENGTH), a, " ")
    ts = sprintf("%s %2d %s", a[2], a[3], a[4])
    last_ts = ts
  }
  gsub(/  +/, " ", rec)
  print ts " " host " local6:info AIXAUDIT: " rec >> logf
  close(logf)
}

/^event|^-----/ { next }
/^[ \t]*$/      { next }
/^[^ \t]/       { emit(pending); pending = $0; next }
                { sub(/^[ \t]+/, ""); emit(pending " | " $0); pending = "" }
END             { emit(pending) }
'
chmod 700 /usr/local/bin/audit2syslog.sh
chown root:system /usr/local/bin/audit2syslog.sh

The script is written this way for three reasons:

  • AIX awk has no fflush(). An earlier version that used it failed with awk: 0602-553 Function fflush is not defined. This version writes each event as soon as its detail line arrives, and close() flushes it.
  • It avoids an audit feedback loop. Calling logger for each event starts a new root process, which is itself a PROC_Execute event, which triggers another logger call, endlessly. A single awk process that runs no commands per event cannot loop.
  • It is safe with log rotation. The file is reopened for every line, so after syslogd rotates messages to messages.0, writes go to the new file.

Step 4: Start audit and keep it on after reboot

audit shutdown
# remove leftovers from any earlier run
ps -ef | grep -E "auditstream|auditpr|audit2syslog" | grep -v grep
audit start
audit query | head -20

su - <testuser> -c id
tail -f /var/log/messages | grep AIXAUDIT

To start audit at boot, check lsitab audit. If nothing is returned, add it:

mkitab "audit:2:once:/usr/sbin/audit start >/dev/console 2>&1"

Step 5: sudo logs every allowed and denied command

Add these lines with visudo:

Defaults syslog=auth
Defaults syslog_goodpri=notice
Defaults syslog_badpri=alert
Defaults syslog_maxlen=2048
Defaults !syslog_pid
Defaults log_allowed, log_denied
  • AIX has no authpriv facility, so auth is set explicitly.
  • syslog_maxlen=2048 stops long commands from being split across several lines.
  • log_allowed and log_denied need sudo 1.9 or newer. On 1.8, remove that line; logging is already on by default.

Check the file with visudo -c.

Step 6: cron job runs and crontab changes

AIX cron writes only to /var/adm/cron/log, so a small forwarder follows that file.

/usr/local/bin/cron2syslog.sh:

#!/usr/bin/ksh
/usr/bin/tail -n 0 -f /var/adm/cron/log | while read -r line; do
  [ -n "$line" ] && /usr/bin/logger -p local5.info -t AIXCRON "$line"
done
chmod 700 /usr/local/bin/cron2syslog.sh
mkitab "cron2sys:2:respawn:/usr/local/bin/cron2syslog.sh"
init q

init restarts the forwarder if it stops and after a reboot. Crontab and at changes (CRON_JobAdd, CRON_JobRemove, AT_JobAdd) come from the cron audit class in Step 2.

Step 7: Binary logs (failedlogin, wtmp, sulog, errpt)

/usr/local/bin/seclog2syslog.sh, run from root’s crontab every 5 minutes:

#!/usr/bin/ksh
STATE=/var/adm/seclog_state; mkdir -p $STATE
fwd() {  # $1=name  $2=command  $3=tag
  $2 > $STATE/$1.new 2>/dev/null
  [ -f $STATE/$1.old ] || touch $STATE/$1.old
  diff $STATE/$1.old $STATE/$1.new | grep '^>' | sed 's/^> //' | \
    while read -r l; do logger -p auth.notice -t "$3" "$l"; done
  mv $STATE/$1.new $STATE/$1.old
}
fwd failedlogin "who /etc/security/failedlogin" AIX_FAILEDLOGIN
fwd wtmp        "who /var/adm/wtmp"              AIX_WTMP
fwd sulog       "cat /var/adm/sulog"             AIX_SULOG
fwd errpt       "errpt"                          AIX_ERRPT

Make it executable with chmod 700, then add this line to root’s crontab with crontab -e:

0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/bin/seclog2syslog.sh

The first run sends the existing history once.

Step 8: Extra sources worth enabling

  • sshd: set SyslogFacility AUTH and LogLevel VERBOSE in /etc/ssh/sshd_config, then restart with stopsrc -s sshd; startsrc -s sshd.
  • Shell history timestamps: set EXTENDED_HISTORY=ON in /etc/environment, and read history with fc -t -l. Users can edit their own history, so treat it as supporting evidence only.
  • FTP: add -l to ftpd in /etc/inetd.conf. Better still, disable FTP and telnet.

Step 9: Show usernames for kernel UID messages

The AIX kernel logs privileged commands with only the numeric user ID, and its message format cannot be changed:

Oct 26 16:39:32 aixprd01 kern:notice unix: The privilege command /usr/lib/boot/bin/bootinfo_chrp, is executed by user with id 338

To find the user quickly on the server:

lsuser -a id ALL | awk '$2=="id=338" {print $1}'

To resolve usernames automatically in Splunk, first add a scripted input on the UF that sends a daily list of user IDs and names. Create /opt/splunkforwarder/etc/apps/aix_users/bin/aix_users.sh:

#!/usr/bin/ksh
/usr/sbin/lsuser -a id ALL | /usr/bin/awk '{ split($2, a, "="); print "user=" $1 " uid=" a[2] }'

Then create /opt/splunkforwarder/etc/apps/aix_users/local/inputs.conf:

[script://./bin/aix_users.sh]
interval   = 86400
index      = aix_os
sourcetype = aix:users
disabled   = false

Make the script executable with chmod 755 and restart the UF.

In Splunk, save this as a daily scheduled search. It builds the lookup, keyed by host because the same ID can belong to different users on different servers:

index=aix_os sourcetype=aix:users earliest=-2d
| stats latest(user) as user by host uid
| outputlookup aix_uid_user.csv

To make the user field appear automatically, add these on the search head.

transforms.conf:

[aix_uid_user]
filename = aix_uid_user.csv

props.conf:

[syslog]
EXTRACT-aix_priv = privilege command (?<priv_cmd>[^,]+), is executed by user with id (?<uid>\d+)
LOOKUP-aix_user  = aix_uid_user host uid OUTPUT user

Sample results

Raw lines in /var/log/messages

Authentication

Oct 26 15:04:01 aixprd01 auth|security:info sshd[5898456]: Accepted password for root from 10.20.30.45 port 51544 ssh2
Oct 26 15:04:12 aixprd01 auth|security:info sshd[6029532]: Failed password for invalid user oracle1 from 10.20.30.77 port 49822 ssh2
Oct 26 15:04:01 aixprd01 local6:info AIXAUDIT: USER_Login root OK Mon Oct 26 15:04:01 2026 sshd 5898456 3801234 | user: root tty: ssh from 10.20.30.45
Oct 26 15:06:02 aixprd01 local6:info AIXAUDIT: USER_SU appadm OK Mon Oct 26 15:06:02 2026 su 6160610 5767392 | root
Oct 26 15:06:30 aixprd01 local6:info AIXAUDIT: USER_SU appadm FAIL Mon Oct 26 15:06:30 2026 su 6160614 5767392 | root
Oct 26 15:11:05 aixprd01 local6:info AIXAUDIT: USER_Create root OK Mon Oct 26 15:11:05 2026 mkuser 6357220 5898456 | testuser01 pgrp=staff home=/home/testuser01

Commands executed and process activity

Oct 26 15:06:45 aixprd01 local6:info AIXAUDIT: PROC_Execute appadm OK Mon Oct 26 15:06:45 2026 ksh 6553834 6160610 | euid: 0 egid: 0 name: /usr/bin/cat /etc/security/passwd
Oct 26 15:09:50 aixprd01 local6:info AIXAUDIT: PROC_Execute appadm FAIL_ACCESS Mon Oct 26 15:09:50 2026 ksh 6815986 5767392 | euid: 203 egid: 1 name: /usr/sbin/shutdown -Fr
Oct 26 15:14:10 aixprd01 local6:info AIXAUDIT: PROC_Kill root OK Mon Oct 26 15:14:10 2026 kill 6881524 5898456 | pid: 4325788 signal: 9
Oct 26 15:15:02 aixprd01 local6:info AIXAUDIT: PROC_SetUserIDs appadm OK Mon Oct 26 15:15:02 2026 su 6947062 5767392 | effect: 0 real: 0 saved: 0 login: 203
Oct 26 16:39:32 aixprd01 kern:notice unix: The privilege command /usr/lib/boot/bin/bootinfo_chrp, is executed by user with id 338

sudo

Oct 26 15:30:11 aixprd01 auth|security:notice sudo:   appadm : TTY=pts/3 ; PWD=/home/appadm ; USER=root ; COMMAND=/usr/bin/cat /etc/security/passwd
Oct 26 15:31:20 aixprd01 auth|security:alert sudo:   appadm : command not allowed ; TTY=pts/3 ; PWD=/home/appadm ; USER=root ; COMMAND=/usr/bin/ksh
Oct 26 15:31:58 aixprd01 auth|security:alert sudo:   devuser1 : user NOT in sudoers ; TTY=pts/5 ; PWD=/home/devuser1 ; USER=root ; COMMAND=/usr/bin/su -

cron

Oct 26 15:33:10 aixprd01 local6:info AIXAUDIT: CRON_JobAdd appadm OK Mon Oct 26 15:33:10 2026 crontab 7536904 6160610 | appadm crontab file updated
Oct 26 15:35:00 aixprd01 local5:info AIXCRON: root : CMD ( /usr/local/bin/backup.sh ) : PID ( 7340290 ) : Mon Oct 26 15:35:00 2026
Oct 26 15:35:42 aixprd01 local5:info AIXCRON: Cron Job with pid: 7340290 Successful

Binary logs

Oct 26 15:15:01 aixprd01 auth|security:notice AIX_FAILEDLOGIN: oracle1 ssh Oct 26 15:04 (10.20.30.77)
Oct 26 15:15:01 aixprd01 auth|security:notice AIX_SULOG: SU 10/26 15:06 - pts/3 appadm-root
Oct 26 15:15:01 aixprd01 auth|security:notice AIX_ERRPT: A924A5FC 1026150926 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED

Exact fields and spacing vary slightly by AIX level and auditpr flags. Check the lines from your own server before building Splunk extractions.

Splunk searches and results

Event count by source over the last 24 hours

index=aix_os host=aixprd01 earliest=-24h
| rex "(?<src_tag>AIXAUDIT|AIXCRON|AIX_\w+|sudo|sshd|unix)"
| stats count by src_tag
| sort - count
src_tag count
AIXAUDIT 41,230
AIXCRON 1,152
sshd 486
sudo 212
AIX_WTMP 96
unix 38
AIX_FAILEDLOGIN 17
AIX_SULOG 9

Audit events with fields extracted

index=aix_os AIXAUDIT
| rex "AIXAUDIT: (?<audit_event>\S+) (?<user>\S+) (?<status>\S+) \w{3} \w{3} +\d+ [\d:]+ \d{4} (?<command>\S+) (?<pid>\d+) (?<ppid>\d+)(?: \| (?<detail>.*))?"
| table _time host audit_event user status command detail
_time host audit_event user status command detail
2026-10-26 15:06:02 aixprd01 USER_SU appadm OK su root
2026-10-26 15:06:30 aixprd01 USER_SU appadm FAIL su root
2026-10-26 15:06:45 aixprd01 PROC_Execute appadm OK ksh euid: 0 egid: 0 name: /usr/bin/cat /etc/security/passwd
2026-10-26 15:33:10 aixprd01 CRON_JobAdd appadm OK crontab appadm crontab file updated

Adjust the rex to match your auditpr field order before saving it in props.conf.

Denied sudo and failed su

index=aix_os ((sudo ("command not allowed" OR "NOT in sudoers" OR "incorrect password")) OR (AIXAUDIT USER_SU FAIL))
| table _time host _raw

Kernel privilege commands with usernames

index=aix_os "privilege command"
| lookup aix_uid_user.csv host uid OUTPUT user
| table _time host user uid priv_cmd
_time host user uid priv_cmd
2026-10-26 16:39:32 aixprd01 appadm 338 /usr/lib/boot/bin/bootinfo_chrp

The counts and usernames above are illustrative, not measurements.

Performance impact

The biggest cost by far is auditing process execution. Everything else in this guide is light.

Component Impact What drives it
Audit PROC_Execute, PROC_Create, PROC_Delete Low to high Number of processes started. Batch-heavy or script-heavy servers can start hundreds of processes per second.
Audit auth, acct, cron, files classes Very low These events are rare.
auditstream, auditpr, audit2syslog.sh pipeline Low One long-running process each. Formatting and writing cost CPU in proportion to event rate.
syslogd writes Very low Small appends to one file.
cron2syslog.sh Very low One logger call per cron log line.
seclog2syslog.sh every 5 min Very low Runs who, errpt and diff; grows with wtmp size.
Splunk UF Low Commonly around 1–2% CPU and roughly 100–200 MB of memory for a single monitored file. Varies by version and volume.
Disk in /var Bounded Capped at about 550 MB by rotate size 50m files 10 (10 rotated files plus the live one).
Splunk license Proportional About 150–250 bytes per audit line. 100,000 events per day is roughly 20 MB per day per server.

Risks to plan for:

  • Busy servers. If the stream pipeline cannot keep up with the event rate, audited processes can be slowed or audit records can be delayed. Test under realistic load before production.
  • Burst delivery. auditpr buffers its output, so on quiet systems events can arrive in small batches. That is a delay in delivery, not lost events.
  • Noise from the forwarders themselves. Each logger call in cron2syslog.sh and seclog2syslog.sh creates a root PROC_Execute audit event. It doesn’t loop, but it adds a little volume.
  • wtmp growth. seclog2syslog.sh re-reads wtmp every 5 minutes, so trim wtmp periodically on long-running servers.

How to measure on your own server:

# CPU and memory of the logging processes
ps -eo pid,pcpu,rss,args | grep -E "auditstream|auditpr|audit2syslog|cron2syslog|splunkd" | grep -v grep

# Overall load before and after enabling audit
vmstat 5 12
topas

# Audit events per minute
grep -c AIXAUDIT /var/log/messages; sleep 60; grep -c AIXAUDIT /var/log/messages

# Log growth in /var
du -sk /var/log/messages*; df -g /var

Ways to reduce the impact:

  1. Keep proc (especially PROC_Execute) for root and admin accounts only, and remove it from default.
  2. Drop PROC_Create and PROC_Delete if you only need executed commands.
  3. Leave CRON_Start and CRON_Finish out; the cron log already records job runs.
  4. Pilot on one server, compare vmstat with and without audit for a day, then tune the classes before rolling out.

Rollback

B=/root/logcfg_backup_<YYYYMMDD>
audit shutdown
cp -p $B/config      /etc/security/audit/config
cp -p $B/streamcmds  /etc/security/audit/streamcmds
cp -p $B/syslog.conf /etc/syslog.conf
cp -p $B/sudoers     /etc/sudoers ; visudo -c
rmitab cron2sys ; init q
pkill -f cron2syslog.sh   # or: ps -ef | grep cron2syslog, then kill <pid>
rmitab audit              # only if added in Step 4
crontab -e                # remove the seclog2syslog line
refresh -s syslogd

Start audit again only if $B/audit_query.txt shows it was running before.

Troubleshooting

Symptom Check
No AIXAUDIT lines Run audit query and `ps -ef
Duplicate lines More than one syslog.conf line writes to /var/log/messages, or two forwarder processes are running.
Too many events Narrow the proc class to admin users only.

Conclusion

With one syslog line, a stream-mode audit pipeline, a few sudoers settings and two small forwarders, AIX security activity lands in a single file as one line per event. The Splunk UF collects it like any other log. Pilot on a non-production host, watch event volume and CPU for a day, tune the audit classes, and then roll out.