Syslogd

Syslogd is the daemon that continuously reads and forwards system messages to the appropriate log
files or users, depending upon the priority of a message and the system facility from which it
originates. The syslogd daemon reads the /etc/syslog.conf to process each and every system message
and direct it to appropriate log file.

The default configuration for syslogd defined to log the system messages ( Warning / Notices / Errors)
within the same host. But sometimes, for ease of administration and troubleshooting, we might
want to configure a single host as as a Centralized loghost and redirect the logs from other solaris
machines to this one centralized host. And configuring such kind of setup is not so difficult. I will
explain you it here.

Configuring Solaris Syslogd for Centralized log Setup

First lets see how the default syslogd configuration works.

/etc/syslog.conf

As I mentioned earlier syslog deamon read /etc/syslog.conf before processing

any system message that triggered within the solaris. And the general
structure of the syslog.conf reflects a set of selectors (facility.level) and actions.
The actions are generally the places to display or log the standard output. For
example, in syslog.conf, there may be several things being sent to /dev/sysmsg
and /var/adm/messages.

A typical entry in syslog.conf looks like this

user.alert /dev/sysmsg

Left side part is facility.level and the right side part is the log destination. For
example Below entry will auth.emerg and kern.crit events are logged in the
same place (in the example below, the /logme file).

auth.emerg; kern.crit /logme

Also, the asterisk can be used to designate all facilities or all levels. This is
useful when you want to log all events of a certain facility or level to one file,
or do something like this:

*.debug; mail.none /var/debuglog

which will send all debug events to the file /var/debuglog except mail.debug
events.

List for Facilities, that can be used in syslog.conf:

user Messages generated by user processes. This is the default

priority for messages from programs or facilities not listed in this file.

kern Messages generated by the kernel.

mail The mail system.

daemon System daemons, such as in.ftpd(1M)

auth The authorization system: login, su, get

lpr The line printer spooling system: lpr(1B), lpc(1B), among

others.

news Designated for the USENET network news system.

cron Designated for cron/at messages generated by systems that

do logging through syslog. The current version of the Solaris Operating
Environment does not use this facility for logging.

audit Designated for audit messages generated by systems that

audit by means of syslog.

local0-7 Designated for local use.

mark For timestamp messages produced internally by syslogd.

 * An asterisk indicates all facilities except for the mark facility.

Examples of Levels that can be used in syslog.conf:

emerg For panic conditions that would normally be broadcast

to all users.

alert For conditions that should be corrected immediately, such

as a corrupted system database.

crit For warnings about critical conditions, such as hard device

errors.

err For other errors.

warning For warning messages.

notice For conditions that are not error conditions, but may
require special handling. A configuration entry with a level value of notice
must appear on a separate line.

info Informational messages.

debug For messages that are normally used only when

debugging a program.

Restart the Syslog daemon to reread the syslog.conf:

# svcadm refresh svc:/system/system-log:default

Running syslogd in debug mode, to troubleshoot logging issues

# svcadm enable svc:/system/system-log:default

# /usr/sbin/syslogd -d
you will see similar output as below if your configuration is valid :

getnets() found 1 addresses, they are: 0.0.0.0.2.2

off & running.
init
amiloghost() testing 129.151.30.223.2.2
cfline(*.err;kern.notice;auth.notice;user.none /dev/console)
cfline(*.err;kern.debug;daemon.notice;mail.crit;user.none /var/adm/messages)
cfline(*.alert;kern.err;daemon.err;user.none operator)
cfline(*.alert;user.none root)
cfline(*.emerg;user.none *)
cfline(auth.notice @loghost)
cfline(mail.debug @loghost)
cfline(user.err /dev/console)
cfline(user.err /var/adm/messages)
cfline(user.alert root, operator)
cfline(user.emerg *)
cfline(daemon.notice /dev/console)
cfline(daemon.notice sean)
cfline(daemon.notice @moog)
cfline(daemon.notice /nsr/logs/messages)
cfline(daemon.notice operator)
cfline(local0.notice /nsr/logs/summary)
cfline(local0.alert root, operator)

If you have an invalid entries you will see something like below

getnets() found 1 addresses, they are: 0.0.0.0.2.2

off & running.
init
amiloghost() testing 129.151.30.223.2.2
cfline(*.err;kern.notice;auth.notice;user.none /dev/console)
cfline(*.err;kern.debug;daemon.notice;mail.crit;user.none /var/adm/messages)
cfline(*.alert;kern.err;daemon.err;user.none operator)
cfline(*.alert;user.none root)
cfline(*.emerg;user.none *)
cfline(auth.notice /var/log/authlog )
syslogd: /var/log/authlog : No such file or directory
logmsg: pri 53, flags 8, from superfreak, msg syslogd: /var/log/authlog : No
such file or directory

Further troubleshooting using the logger command

The /usr/bin/logger command can be used for troubleshooting and manual

logging. For example, to send a message of priority auth.notice, use the syntax:

# logger -p daemon.notice test

A tail of /var/adm/messages will show the test message:

# tail /var/adm/messages

Mar 1 17:01:52 persia gurkulsolaris: [ID 702911 daemon.notice] test

Now we will see how to configure the Centralized Log

Setup:
All you need to do here is send the syslog messages to a remote host by
specifying that host in the action field as shown below:

auth.notice @loghost

or
auth.notice @gurkullogserver

Note : You need to make sure that the hostname is referring the IP of the
centralized log server. Just for verification you can run the command

# ping -s loghost

And make the modifications necessary to the syslog.conf file on the centralized
system you want to log. And please remember that the facility you log to the
remote host will be logged according to how the action field is set up on the
remote system. You must restart syslogd for changes to take effect.

just in case if you want to setup multiple targets to the system messages

If you want to sent the Authentication Messages for both the loghost and local
authlog, you can configure multiple targets by using either one below format.

auth.notice ifdef(`LOGHOST, /var/log/authlog, @loghost)

Or

auth.notice /var/log/authlog
auth.notice @loghost

Troubleshoot Syslog issues in Centralized log environment

You can use the same troubleshooting mentioned about, and in addition to that
you can also use the snoop tool to check the syslog traffic between your solaris
hosts and centralized log server , while running a logger command. That will tell
you whether the local syslog really trying to reach the centralized syslog server
or not.
#snoop udp between myserver gurkullogserver
Using device /dev/hme (promiscuous mode)
myserver -> gurkullogserver SYSLOG C port=35725 <37>May 1 10:42:33
myserver -> gurkullogserver SYSLOG C port=35725 <34>May 1 10:45:12

How do I send all information in /var/adm/message file to a remote system?

You can simply edit the /etc/syslog.conf file and, wherever /var/adm/messages appears,
duplicate the line and replace /var/adm/messages by @remoteSystem with remoteSystem being
the IP address or hostname of the remote server where to send the logs.

eg:

before:

...

*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages

...

after:

...

*.err;kern.debug;daemon.notice;mail.crit /var/adm/messages

*.err;kern.debug;daemon.notice;mail.crit @jaylogserver

...

Restart syslogd for the change to be taken into account:

# svcadm restart system-log

Note that the remote server must be configured to accept remote messages. If on Solaris
too, that would be done with this command:

# svccfg -s system-log setprop config/log_from_remote = true

# svcadm restart system-log

Difference between /var/log/syslog and /var/adm/messages

Most commonly they are not the same. To see what kind of messages go to /var/adm/messages
and which go to /var/log/syslog, check /etc/syslog.conf