I'm Kevin. I live in Salem, Mass with my wife and little boy and I build software.

Drupal's Syslog module is one of the core modules that I find is often overlooked. On a high traffic site, it's generally a good idea to reduce the db traffic as much as possible. Besides, since the database logging doesn't rotate/archive logs, it is only marginally helpful. So, while you need watchdog, it can quickly bite you in the... sorry, bad pun, but you get the idea. The good news is that we can still log all of these helpful messages in a flat file using Syslog and do some insanely cool reporting using Splunk.

A proper configuration of Syslog can really come to your rescue here. The Syslog module uses PHP's appropriately named syslog() function to log Drupal messages to the filesystem. You can begin by disabling Drupal's "Database Logging" module and enabling the Syslog module.

The next step is to separate the noise from your normal message log. The Syslog module offers a minimal configuration that lets you specify the syslog facility code. This code lets your syslog daemon know how to channel the requests. In linux, for example, you can specify "Local 0" through "Local 7". Then in your syslog.conf file, you can specify which file these particular messages get saved to. For example, if you choose "LOG_LOCAL6" in the Syslog module settings, you can then segregate your Drupal messages to their own file below (the last line is the only really important one):

/etc/syslog.conf

Log all kernel messages to the console.

Logging much else clutters up the screen.

kern.* /dev/console

Log anything (except mail) of level info or higher.

Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none /var/log/messages

The authpriv file has restricted access.

authpriv.* /var/log/secure

Log all the mail messages in one place.

mail.* -/var/log/maillog

Log cron stuff

cron.* /var/log/cron

Everybody gets emergency messages

*.emerg *

Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

Save boot messages also to boot.log

local7.* /var/log/boot.log

Save the Drupal log separately - using LOG_LOCAL6

in Drupal's Syslog module settings.

local6.* /var/log/drupal

Restart your syslog and you should immediately see messages getting dumped into /var/log/drupal.

Reporting with Splunk

So, now that you have the messages out of the database and into a file, how do you make sense of them? Try out Splunk. Splunk is a really awesome tool that you can use to get the most from your new Drupal logs. In addition, you can use it to see all of your logs in one place. If your log is in a structured format, Splunk can index it and let you search and report on it. I won't go into too much detail about Splunk other than to say that the search mechanism is really amazing and even lets you run scheduled reports and send alerts. It is really easy to install and even easier to index new logs.

With a small amount of work, you will end up with a more efficient Drupal site and a much more usable reporting system. Just give it a try and if you are anything like me, you'll waste a whole afternoon indexing all kinds of weird stuff and creating every report you can think of :)

-- Edit --

It's also worth noting that you can easily create your own log formats by overriding theme_syslog_format().