Using Syslog Module and Splunk to Rock Your Reporting World
Drupal's Syslog module is one of the core modules that I find is often overlooked. On a high traffic site, it's generally a good idea to reduce the db traffic as much as possible. Besides, since the database logging doesn't rotate/archive logs, it is only marginally helpful. So, while you need watchdog, it can quickly bite you in the... sorry, bad pun, but you get the idea. The good news is that we can still log all of these helpful messages in a flat file using Syslog and do some insanely cool reporting using Splunk.
Keeping an eye on memory usage
So, while working on a project recently we were faced with the reality that Drupal was using too much memory for our server to handle. The culprit was an AJAX refresh script that helped us to keep the page content up-to-date without refreshing. We had spent a good amount of time tuning the amount of processing and bandwidth used by the module, but the Drupal full bootstrap was just killing us with a 10 second refresh rate.
The solution we settled on was to not bootstrap Drupal if it wasn't necessary. We implemented a preliminary PHP script that the AJAX request was first passed to. This script opened a MySQL connection and ran a single query to check for updated content. If there was none, the script returned empty JSON. If content was found, then Drupal was fully bootstrapped to load the content and return a full JSON structure with the new content.
The results shouldn't be surprising, but still blew me away. Using the PHP function memory_get_peak_usage() we were able to see just how much memory the scripts were using at the point where the JSON was returned. When new content was found and Drupal bootstrapped PHP used 16961212 bytes (16.18 MB) of memory, but when there was no new content, the intermediate script only used 86968 bytes (84.9 KB) of memory.
Read more for the code...