Linux Syslog with cisco

syslog

syslog is a utility for tracking and logging all manner of system messages from the merely informational to the extremely critical. Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle:

• The first describes the function (facility) of the application that generated it. For example, applications such as mail and cron generate messages with easily identifiable facilities named mail and cron.

• The second describes the degree of severity of the message. There are eight in all and they are listed in

  Severity Level	Keyword	Description	0	emergencies	System unusable
  1	alerts	Immediate action required
  2	critical	Critical condition
  3	errors	Error conditions
  4	warnings	Warning conditions
  5	notifications	Normal but significant conditions
  6	informational	Informational messages
  7	debugging	Debugging messages

You can configure syslog‘s /etc/syslog.conf configuration file to place messages of differing severity and facilities in different files. This procedure will be covered next.

The /etc/syslog.conf File

The files to which syslog writes each type of message received is set in the /etc/syslog.conf configuration file. This file consists of two columns: The first lists the facilities and severity of messages to expect, and the second lists the files to which they should be logged. By default, Red Hat/Fedora’s /etc/syslog.conf file is configured to put most of the messages in the file /var/log/messages. Here is a sample:

     .info;mail.none;authpriv.none;cron.none           /var/log/messages

In this case, all messages of severity “info” and above are logged, but none from the mail, cron, or authentication facilities/subsystems. You can make this logging even more sensitive by replacing the line above with one that captures all messages from debug severity and above in the /var/log/messages file. This may be more suitable for troubleshooting:

     *.debug                     /var/log/messages

Certain applications will additionally log to their own application specific log files and directories independent of the syslog.conf file. Here are some common examples:

Files

     /var/log/maillog           : Mail
     /var/log/httpd/access_log  : Apache web server page access logs

Directories

     /var/log
     /var/log/samba                : Samba messages
     /var/log/mrtg                 : MRTG messages
     /var/log/httpd                : Apache webserver messages

Activating Changes to the syslog Configuration File

Changes to /etc/syslog.conf will not take effect until you restart syslog. Issue this command to do so:

     [root@bigboy tmp]# service syslog restart

How to View New Log Entries as They Happen

If you want to get new log entries to scroll on the screen as they occur, you can use this command:

     [root@bigboy tmp]# tail -f /var/log/messages

Similar commands can be applied to all log files. This is probably one of the best troubleshooting tools available in Linux. Another good command to use apart from tail is grep. grep will help you search for all occurrences of a string in a log file; you can pipe it through the more command so that you only get one screen at a time. Here is an example:

     [root@bigboy tmp]# grep string /var/log/messages | more

You can also just use the plain old more command to see one screen at a time of the entire log file without filtering with grep. Here is an example:

     [root@bigboy tmp]# more /var/log/messages

Logging syslog Messages to a Remote Linux Server

Logging your system messages to a remote server is a good security practice. With all servers logging to a central syslog server, it becomes easier to correlate events across your company. It also makes covering up mistakes or malicious activities harder because the purposeful deletion of log files on a server cannot simultaneously occur on your logging server, especially if you restrict the user access to the logging server.

Configuring the Linux syslog Server

By default, syslog doesn’t expect to receive messages from remote clients. Here’s how to configure your Linux server to start listening for these messages.

As we saw previously, syslog checks its /etc/syslog.conf file to determine the expected names and locations of the log files it should create. It also checks the file /etc/sysconfig/syslog to determine the various modes in which it should operate. syslog will not listen for remote messages unless the SYSLOGD_OPTIONS variable in this file has an -r included in it:

     # Options to syslogd
     # -m 0 disables 'MARK' messages.
     # -r enables logging from remote machines
     # -x disables DNS lookups on messages received with -r
     # See syslogd(8) for more details
     SYSLOGD_OPTIONS="-m 0 -r"
     # Options to klogd
     # -2 prints all kernel oops messages twice; once for klogd to decode,
     and
     #    once for processing with 'ksymoops'
     # -x disables all klogd processing of oops messages entirely
     # See klogd(8) for more details
     KLOGD_OPTIONS="-2"

You have to restart syslog on the server for the changes to take effect. The server will now start to listen on UDP port 514, which you can verify using either one of the following netstat command variations:

     [root@bigboy tmp]# netstat -a | grep syslog
     udp        0      0 *:syslog                *:*
     [root@bigboy tmp]# netstat -an | grep 514
     udp        0      0 0.0.0.0:514             0.0.0.0:*
     [root@bigboy tmp]#

Configuring the Linux Client

The syslog server is now expecting to receive syslog messages. You have to configure your remote Linux client to send messages to it. This is done by editing the /etc/hosts file on the Linux client named smallfry. Here are the steps:

  IP-address    fully-qualified-domain-name    hostname    "loghost"

  192.168.1.100   bigboy.my-web-site.org    bigboy    loghost

*.debug                     @loghost
*.debug                     /var/log/messages

You have now configured all debug messages and higher to be logged to both server bigboy (“loghost”) and the local file /var/log/messages. Remember to restart syslog to get the remote logging started.

You can now test to make sure that the syslog server is receiving the messages with a simple test, such as restarting the lpd printer daemon and making sure the remote server sees the messages.

Linux Client

     [root@smallfry tmp]# service lpd restart
     Stopping lpd: [ OK ]
     Starting lpd: [ OK ]
     [root@smallfry tmp]#

Linux Server

     [root@bigboy tmp]# tail /var/log/messages
     ...
     ...
     Apr 11 22:09:35 smallfry lpd: lpd shutdown succeeded
     Apr 11 22:09:39 smallfry lpd: lpd startup succeeded
     ...
     ...
     [root@bigboy tmp]#

syslog Configuration and Cisco Network Devices

syslog reserves facilities local0 through local7 for log messages received from remote servers and network devices. Routers, switches, firewalls, and load balancerseach logging with a different facilitycan each have their own log files for easy troubleshooting. Appendix IV has examples of how to configure syslog to do this with Cisco devices using separate log files for the routers, switches, PIX firewalls, CSS load balancers, and LocalDirectors.

syslog and Firewalls

syslog listens by default on UDP port 514. If you are logging to a remote syslog server via a firewall, you have to allow traffic on this port to pass through the security device. syslog messages usually have UDP port 514 for both their source and destination UDP ports.

logrotate

The Linux utility logrotate renames and reuses system error log files on a periodic basis so that they don’t occupy excessive disk space.

The /etc/logrotate.conf File

The /etc/logrotate.conf file is logrotate‘s general configuration file in which you can specify the frequency with which the files are reused:

• You can specify either a weekly or daily rotation parameter. In the case below, the weekly option is commented out with a #, allowing daily updates.

• The rotate parameter specifies the number of copies of log files logrotate will maintain. In the case below, the 4 copy option is commented out with a #, while allowing 7 copies.

• The create parameter creates a new log file after each rotation.

Therefore, our sample configuration file will create daily archives of all the logfiles and store them for seven days. The files will have the following names, with logfile the current active version:

     logfile
     logfile.0
     logfile.1
     logfile.2
     logfile.3
     logfile.4
     logfile.5
     logfile.6

Sample Contents of /etc/logrotate.conf

     # rotate log files weekly
     #weekly

     # rotate log files daily
     daily

     # keep 4 weeks worth of backlogs
     #rotate 4

     # keep 7 days worth of backlogs
     rotate 7

     # create new (empty) log files after rotating old ones
     create

The /etc/logrotate.d Directory

Most Linux applications that use syslog put an additional configuration file in this directory to specify the names of the log files to be rotated. It is a good practice to verify that all new applications that you want to use the syslog log have configuration files in this directory. Here are some sample files that define the specific files to be rotated for each application.

The /etc/logrotate.d/syslog File (for General System Logging)

     /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler
     /var/log/boot.log /var/log/cron {
         sharedscripts
         postrotate
         /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2>
     /dev/null || true
         endscript
     }

The /etc/logrotate.d/apache File (for Apache)

     /var/log/httpd/access_log /var/log/httpd/agent_log
     /var/log/httpd/error_log /var/log/httpd/referer_log {
         missingok
         sharedscripts
         postrotate
         /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null
     || true
         endscript
     }

The /etc/logrotate.d/samba File (for Samba)

     /var/log/samba/*.log {
         notifempty
         missingok
         sharedscripts
         copytruncate
         postrotate
         /bin/kill -HUP `cat /var/lock/samba/*.pid 2> /dev/null` 2>
     /dev/null || true
         endscript
     }

Activating logrotate

The logrotate settings in the last section will not take effect until you issue the following command:

     [root@bigboy tmp]# logrotate -f

If you want logrotate to reload only a specific configuration file, and not all of them, issue the logrotate command with just that filename as the argument:

     [root@bigboy tmp]# logrotate -f /etc/logrotate.d/syslog

Compressing Your Log Files

On busy Web sites the size of your log files can become quite large. Compression can be activated by editing the logrotate.conf file and adding the compress option.

     #
     # File: /etc/logrotate.conf
     #

     # Activate log compression
     compress

The log files will then start to become archived with the gzip utility, each file having a .gz extension.

     [root@bigboy tmp]# ls /var/log/messages*
     /var/log/messages      /var/log/messages.1.gz /var/log/messages.2.gz
     /var/log/messages.3.gz /var/log/messages.4.gz /var/log/messages.5.gz
     /var/log/messages.6.gz /var/log/messages.7.gz
     [root@bigboy tmp]#

Viewing the contents of the files still remains easy because the zcat command can quickly output the contents to the screen. Use the command with the compressed file’s name as the argument:

     [root@bigboy tmp]# zcat /var/log/messages.1.gz
     ...
     ...
     Nov 15 04:08:02 bigboy httpd: httpd shutdown succeeded
     Nov 15 04:08:04 bigboy httpd: httpd startup succeeded
     Nov 15 04:08:05 bigboy sendmail[6003]: iACFMLHZ023165:
     to=<tvaughan@clematis4spiders.info>, delay=2+20:45:44,
     xdelay=00:00:02, mailer=esmtp, pri=6388168,
     relay=www.clematis4spiders.info. [222.134.66.34], dsn=4.0.0,
     stat=Deferred: Connection refused by www.clematis4spiders.info.
     [root@bigboy tmp]#

syslog Configuration and Cisco Devices

syslog reserves facilities local0 tHRough local7 for log messages received from remote servers and network devices. Routers, switches, firewalls, and load balancers each logging with a different facility can each have their own log files for easy troubleshooting. This appendix will show you how to have a different log file for each class of device. All the network device configuration examples that follow log to the remote Linux logging server 192.168.1.100. Remember, if you have a large data center, you may also want to switch off all logging to /var/log/messages for the home/SOHO environment

     service timestamps log datetime localtime
     no logging console
     no logging monitor
     logging 192.168.1.100

Catalyst CAT Switches Running CATOS

By default Cisco switches also send syslog messages to their logging server with a default facility of local7. Don’t change this facility either, therefore making routers and switches log to the same file.

     set logging server enable
     set logging server 192.168.1.100
     set logging level all 5
     set logging server severity 6

Cisco Local Director

Local Directors use the syslog output command to set the logging facility and severity. The value provided must be in the format FF.SS (facility.severity) using the numbering scheme in

This example uses facility local4 and the logging debugging messages from

     syslog output 20.7
     no syslog console
     syslog host 192.168.1.100

Cisco PIX Firewalls

PIX firewalls use the numbering scheme in to determine their logging facilities.

This configuration example assumes that the logging server is connected on the side of the “inside” protected interface. It sends log messages to facility local3 with a severity level of 5 (Notification) set by the logging trap command.

     logging on
     logging standby
     logging timestamp
     logging trap notifications
     logging facility 19
     logging host inside 192.168.1.100

Cisco CSS11000 (Arrowpoints)

The configuration for the Cisco CSS11000 load balancer series is more straightforward. You specify the facility with an intuitive number using the logging host command and set the severity with the logging subsystem command. This example shows the CSS11000 logging facility local6 and severity level 6 (Informational):

     logging host 192.168.1.100 facility 6
     set logging subsystem all info-6

     logging commands enable

The Sample Cisco syslog.conf File

     #
     # All LOCAL3 messages (debug and above) go to the firewall file
     ciscofw
     #

     local3.debug /var/log/cisco/ciscofw
     #
     # All LOCAL4 messages (debug and above) go to the Local Director file
     ciscold
     #
     local4.debug /var/log/cisco/ciscold

     #
     # All LOCAL6 messages (debug and above) go to the CSS file ciscocss
     #
     local6.debug /var/log/cisco/ciscocss

     #
     # All LOCAL7 messages (debug and above) go to the ciscoacl
     # This includes ACL logs which are logged at severity debug
     #
     local7.debug /var/log/cisco/ciscoacl

     #
     # LOCAL7 messages (notice and above) go to the ciscoinfo
     # This excludes ACL logs which are logged at severity debug
     #
     local7.notice /var/log/cisco/ciscoinfo