Search This Blog

Saturday, November 15, 2008

UNIX System Log (syslog)

Many versions of UNIX provide a general-purpose logging facility called syslog, originally developed at the University of California at Berkeley for the Berkeley sendmail program. Since then, syslog has been ported to several System V-based systems, and is now widely available. The uses of syslog have similarly been expanded.

syslog is a host-configurable, uniform system logging facility. The system uses a centralized system logging process that runs the program /etc/syslogd or /etc/syslog. Individual programs that need to have information logged send the information to syslog. The messages can then be logged to various files, devices, or computers, depending on the sender of the message and its severity.

Any program can generate a syslog log message. Each message consists of four parts:

  • Program name

  • Facility

  • Priority

  • Log message itself

For example, the message:

login: Root LOGIN REFUSED on ttya

is a log message generated by the login program. It means that somebody tried to log into an unsecure terminal as root. The messages's facility (authorization) and error level (critical error) are not shown.

The syslog facilities are summarized in Table 1, Not all facilities are present on all versions of UNIX.

Table 1: syslog Facilities

Name

Facility

kern

Kernel

user

Regular user processes

mail

Mail system

lpr

Line printer system

auth

Authorization system, or programs that ask for user names and passwords (login, su, getty, ftpd, etc.)

daemon

Other system daemons

news

News subsystem

uucp

UUCP subsystem

local0... local7

Reserved for site-specific use

mark

A timestamp facility that sends out a message every 20 minutes

The syslog priorities are summarized in Table 2

Table 10.2: syslog Priorities

Priority

Meaning

emerg

Emergency condition, such as an imminent system crash, usually broadcast to all users

alert

Condition that should be corrected immediately, such as a corrupted system database

crit

Critical condition, such as a hardware error

err

Ordinary error

warning

Warning

notice

Condition that is not an error, but possibly should be handled in a special way

info

Informational message

debug

Messages that are used when debugging programs

none

Do not send messages from the indicated facility to the selected file. For example, specifying *.debug;mail.none sends all messages except mail messages to the selected file.

When syslogd starts up, it reads its configuration file, usually /etc/syslog.conf, to determine what kinds of events it should log and where they should be logged. syslogd then listens for log messages from three sources, shown in below Table 3

Table 3: Log Message Sources

Source

Meaning

/dev/klog

Special device, used to read messages generated by the kernel

/dev/log

UNIX domain socket, used to read messages generated by processes running on the local machine

UDP port 514

Internet domain socket, used to read messages generated over the local area network from other machines

Overview of Syslog

System administrators have to deal with lots of different kinds of messages from a plethora of subsystems within each system, and usually lots of systems as well. For example, an FTP server might report every connection it gets. The kernel might report hardware failures on a disk drive. A DNS server might report usage statistics at regular intervals.

Some of these messages need to be brought to a system administrator's attention immediately. And it may not be just any system administrator – there may be a particular system administrator who deals with a particular kind of message. Other messages just need to be recorded for future reference if there is a problem. Still others may need to have information extracted from them by an automated process that generates monthly reports.

To deal with these messages, most Unix systems have a facility called "Syslog." It is generally based on a daemon called “Syslogd” Syslogd listens for messages on a Unix domain socket named /dev/log. Based on classification information in the messages and its configuration file (usually /etc/syslog.conf), Syslogd routes them in various ways. Some of the popular routings are:

  • Write to the system console
  • Mail to a specific user
  • Write to a log file
  • Pass to another daemon
  • Discard

Syslogd can also handle messages from other systems. It listens on the syslog UDP port as well as the local socket for messages.

Syslog can handle messages from the kernel itself. But the kernel doesn't write to /dev/log; rather, another daemon (sometimes called “Klogd”) extracts messages from the kernel and passes them on to Syslog as any other process would (and it properly identifies them as messages from the kernel).

Syslog can even handle messages that the kernel issued before Syslogd or Klogd was running. A Linux kernel, for example, stores startup messages in a kernel message ring and they are normally still there when Klogd later starts up. Assuming Syslogd is running by the time Klogd starts, Klogd then passes everything in the message ring to it.

In order to classify messages for disposition, Syslog requires any process that submits a message to it to provide two pieces of classification information with it:

facility
This identifies who submitted the message. There are a small number of facilities defined. The kernel, the mail subsystem, and an FTP server are examples of recognized facilities. For the complete list, See syslog; vsyslog. Keep in mind that these are essentially arbitrary classifications. "Mail subsystem" doesn't have any more meaning than the system administrator gives to it.
priority
This tells how important the content of the message is. Examples of defined priority values are: debug, informational, warning, critical. For the complete list, See syslog; vsyslog. Except for the fact that the priorities have a defined order, the meaning of each of these priorities is entirely determined by the system administrator.

A “facility/priority” is a number that indicates both the facility and the priority.

Warning: This terminology is not universal. Some people use “level” to refer to the priority and “priority” to refer to the combination of facility and priority. A Linux kernel has a concept of a message “level,” which corresponds both to a Syslog priority and to a Syslog facility/priority (It can be both because the facility code for the kernel is zero, and that makes priority and facility/priority the same value).

Logging everything everywhere

Disks are cheap these days. Sites with sufficient resources and adequately trained personnel sometimes choose to log everything that might possibly be useful, and log it in many different places. For example, clients can create their own log files of syslog events, and also send all logging messages to several different logging hosts - possibly on different networks.

The advantage of logging in multiple places is that it makes an attacker's attempts at erasing any evidence of his presence much more difficult. On the other hand, multiple log files will not do you any good if they are never examined. Furthermore, if they are never pruned, they may grow so large that they will shut down your computers.

Syslog Messages

The following tables[11] summarize some typical messages available on various versions of UNIX.:

[11] A similar list is not available for System V UNIX, because syslog is part of the Berkeley UNIX offering. Companies that sell syslog with their System V offerings may or may not have modified the additional programs in their operating systems to allow them to use the syslog logging facility.

Table 10.4: Typical Critical Messages

Program

Message

Meaning

halt

halted by

used the /etc/halt command to shut down the system.

login

ROOT LOGIN REFUSED ON [FROM ]

root tried to log onto a terminal that is not secure.

login

REPEATED LOGIN FAILURES ON [FROM ]

Somebody tried to log in as and supplied a bad password more than five times.

reboot

rebooted by

rebooted the system with the /etc/reboot command.

su

BAD SU on

Somebody tried to su to the superuser and did not supply the correct password.

shutdown

reboot, halt, or shutdown by on

used the /etc/shutdown command to reboot, halt, or shut down the system.

Other critical conditions that might be present might include messages about full filesystems, device failures, or network problems.

Table 10.5: Typical Info Messages

Program

Message

Meaning

date

date set by

changed the system date.

login

ROOT LOGIN [FROM ]

root logged in.

su

on

used the su command to become the superuser.

getty

/bin/getty was unable to open .

NOTE: For security reasons, some information should never be logged. For example, although you should log failed password attempts, you should not log the password that was used in the failed attempt. Users frequently mistype their own passwords, and logging these mistyped passwords would help a computer cracker break into a user's account. Some system administrators believe that the account name should also not be logged on failed login attempts - especially when the account typed by the user is nonexistent. The reason is that users occasionally type their passwords when they are prompted for their usernames. If invalid accounts are logged, then it might be possible for an attacker to use those logs to infer people's passwords.

You may want to insert syslog calls into your own programs to record information of importance. Third-party software also often has a capability to send log messages into the syslog if configured correctly. For example, Xyplex terminal servers and Cisco routers both can log information to a network syslog daemon; Usenet news and POP mail servers also log information.

If you are writing shell scripts, you can also log to syslog. Usually, systems with syslog come with the logger command. To log a warning message about a user trying to execute a shell file with invalid parameters, you might include:

logger -t ThisProg -p user.notice "Called without required # of parameters"

NOTE: Prior to 1995, many versions of the syslog library call did not properly check their inputs to be certain that the data would fit into the function's internal buffers. Thus, many programs could be coerced to accept input to write arbitrary data over their stacks, leading to potential compromise. Be certain that you are running software using a version of syslog that does not have this vulnerability.

Beware false log entries

The UNIX syslog facility allows any user to create log entries. This capability opens up the possibility for false data to be entered into your logs. An interesting story of such logging was given to us by Alec Muffet:

A friend of mine - a UNIX sysadmin - enrolled as a mature student at a local polytechnic in order to secure the degree which had been eluding him for the past four years.

One of the other students on his Computer Science course was an obnoxious geek user who was shoulder surfing people and generally making a nuisance of himself, and so my friend determined to take revenge.

The site was running an early version of Ultrix on an 11/750, but the local operations staff were somewhat paranoid about security, had removed world execute from "su" and left it group-execute to those in the wheel group, or similar; in short, only the sysadmin staff should have execute access for su.

Hence, the operations staff were somewhat worried to see messages with the following scrolling up the console:

  BAD SU: geekuser ON ttyp4 AT 11:05:20
 BAD SU: geekuser ON ttyp4 AT 11:05:24
 BAD SU: geekuser ON ttyp4 AT 11:05:29
 BAD SU: geekuser ON ttyp4 AT 11:05:36
 ...

When the console eventually displayed:

  SU: geekuser ON ttyp4 AT 11:06:10

all hell broke loose: the operations staff panicked at the thought of an undergrad running around the system as root and pulled the plug (!) on the machine. The system administrator came into the terminal room, grabbed the geekuser, took him away and shouted at him for half an hour, asking (a) why was he hacking, (b) how was he managing to execute su and (c) how he had guessed the root password?

Nobody had noticed my friend in the corner of the room, quietly running a script which periodically issued the following command, redirected into /dev/console, which was world-writable:

  echo BAD SU: geekuser ON ttyp4 AT `date` 

The moral of course is that you shouldn't panic, and that you should treat your audit trail with suspicion.

No comments: