WHM/cPanel Log Processing

Copyright 2010 cPanel, Inc. Revision 3 Revision 4 Revision 5 Revision 6 Revision 7 Revision History 2010-01-05 2010-01-05 2010-01-07 2010-01-10 2010-01-11 Abstract This document details the WHM/cPanel log processing related to Apache and WHM Accounts. Particular attention is paid to version 11.25.0 which introduced interface and behavioral changes. Versions >= 11.25.0 dn kp kp kp kp

Table of Contents
1. Just the Facts ..................................................................................................................................... 1 2. Overview ........................................................................................................................................... 2 2.1. cpanellogd ......................................................................................................................... 2 2.2. Version 11.25.0 ........................................................................................................................ 2 3. How Log Processing Works ................................................................................................................. 3 3.1. Statistics Generation ................................................................................................................. 3 3.2. Rotation & Retention ................................................................................................................ 5 3.3. Synchronicity ........................................................................................................................... 6 4. Configuration ..................................................................................................................................... 7 4.1. Cycle Frequency ...................................................................................................................... 7 4.2. Log Rotation ........................................................................................................................... 8 4.3. WHM Tweak Settings ............................................................................................................. 10 4.4. cPanel Options .................................................................................................................... 11 4.5. Flag Files .............................................................................................................................. 12 4.6. Other Logs ............................................................................................................................ 12 5. Troubleshooting ................................................................................................................................ 12 6. Known Issues and Caveats ................................................................................................................. 13 6.1. Apache Restarts ...................................................................................................................... 13 7. Conclusion ....................................................................................................................................... 13 Glossary .............................................................................................................................................. 13

1. Just the Facts

Q: A: Why is Apache restarting so often? Depending upon the log processing configuration cpanellogd will restart Apache to preserve data integrity within the account domain and bytes logs. See Section 3.1.1, Apache Restart Requirement How can I stop or lower the frequency of Apache restarts? There are several ways to do this:

Q: A:

WHM/cPanel Log Processing

Reduce the frequency of log processing Use the following values for the Schedule Configuration section of the Statistics Software Configuration interface within WHM: Process log files every 24 hours. Process bandwidth every 8 hours. Modify the timestamps of the files in /var/cpanel/lastrun. See "Contents of /var/cpanel/lastrun" for more information.. Change the following: Uncheck the Tweak Setting labeled Delete each domain's access logs after stats run

Caution
Leaving the log files in place may cause disk utilization issues, especially on systems that cannot handle files larger than 2 GB Disable log archiving. This is something the end user may configure. Configure Log Processing Blackout hours ( see Black-out Hours ) to prevent log processing during busy traffic periods.

2. Overview
2.1. cpanellogd
WHM/cPanel processing related to the domain logs and bandwidth logs for Accounts, as well as service level logs for cPanel and Apache managed by the cpanellogd daemon. cpanellogd, in all versions of WHM/cPanel, is responsible for tracking log file size, rotation and parsing frequency. Any utility available to affect cpanellogd's behavior is exposed in the web interface as a value field, checkbox, or note for custom configuration.

2.2. Version 11.25.0

The Web Host Manager and cPanel suite version 11.25.0 processes statistics in a different, more robust way than past versions. At a glance one will find two immediate differences: 1. A new configuration setting in Statistics Software Configuration within the WHM interface. 2. Periodic entries in the Apache log that Graceful restart[s] were requested and performed. These differences highlight a new approach to statistics processing that aims to reduce processing time, CPU demand, and increase scalability.

2.2.1. Interface
Statistics processing is scheduled in WHM via Statistics Software Configuration under Server Configuration in the sidebar. Previous versions of the WHM interface had a Statistics Frequency hourly value field only. Now the Schedule Configuration subsection has two hourly value fields, Log Processing Frequency and Bandwidth Processing Frequency. The differences and affect of these values are detailed below.

WHM/cPanel Log Processing

Note
These value fields are mirrored in 'Main >> Server Configuration >> Tweak Settings >> Stats and Logs.' However, since these value fields are most appropriately placed in the "Schedule Configuration" interface, it should be inferred such mirroring is only present for legacy purposes and is likely to be deprecated.

Note
More interface changes are expected as respective functionality is refactored in future versions.

2.2.2. Apache Behavior

Log processing is determined by the combination of several values within cPanel. Optimal, balanced performance from v11.25.0 can often result in Apache restarts approximately every 15 mins. Details on frequency and the affecting factors are outlined below.

3. How Log Processing Works

cPanel will manage logs related to statistical information as well as the Apache logs. This consists of statistical log parsing, rotation, archiving and deletion. All other system or sub-system logs (i.e., messages, maillog, etc.) are beyond the of scope for this document.

3.1. Statistics Generation

cpanellogd triggers the log parsers which, in turn, report data for Daily Process Log, View Bandwidth Usage and the various report engines available at the account & domain level. Triggers are governed, in part1, by the values set in WHM's Statistics Software Configuration. Hourly values defined in Log and Bandwidth Frequency fields do not determine the "run" frequency of cpanellogd, but instead determines the threshold of time that respective log files are allowed to lapse between parsing. cpanellogd checks the Logs and Bandwidth files about every 15 minutes. More precisely, cpanellogd will attempt to evaluate a set of logs 500 seconds (8'20") proceeding the last evaluation and parse-cycle's completion; alternating between Log (domain) files and Bandwidth files. See Example 1, cpanellogd Check Times. If any file has not been parsed since "X hourly value", the file is queued for parsing.

Example 1. cpanellogd Check Times

Server blue.example.com has the following configuration for Bandwidth and Log processing frequency: Bandwidth: Log: 2 Hrs. 6 Hrs.

At 14:00:00 UTC cpanellogd checks whether it needs to process any Bandwidth files. At 14:08:20 UTC cpanellogd checks whether it needs to process any Log files. The next check at 14:16:40 UTC, where it checks for Bandwidth files that need processed, is approximately 15 minutes since the last check of the Bandwidth files.
1

Additional cpanellogd governance is found in "Schedule Configuration >> Statistics Schedule," "Tweak Settings" and '/var/cpanel/cpanel.config'. See the Configuration portion of this document for more detail.

WHM/cPanel Log Processing

"X" is the respective value of Log Processing Frequency or Bandwidth Processing Frequency value fields; defined in Main >> Server Configuration >> Statistics Software Configuration >> Schedule Configuration. Once all appropriate items are queued, the parsing cycle begins. During a parse cycle 5 key events occur: 1. A zero length file is made in /var/cpanel/lastrun/ indicating last processing date time. See "Contents of /var/cpanel/lastrun" for more details. 2. All queued log files are renamed with a '.bkup' extension. 3. Apache is gracefully restarted. 4. After the Apache restart, cpanellogd forks a child process for the necessary parser. 5. Once the parser completes operation on the .bkup file, rotation needs are determined: if the give log is scheduled for rotation, ${logfile}.bkup is prepended to the active log file, else if ${logfile} is not scheduled for rotation, no action is taken. 6. The ${logfile}.bkup files are discarded. Contents of /var/cpanel/lastrun Each cPanel account has a corresponding directory in /var/cpanel/lastrun. Within the account directory are two files used to indicate the last time cpanellogd processed log files for the account. Each file corresponds to type of log processed.

Table 1. Last Run Flag Files

File bandwidth stats Log Type bytes_log The Apache access logs in /usr/local/apache/domlogs

3.1.1. Apache Restart Requirement

As noted above, an Apache restart is part of the parsing process. If none of the items queued for processing need to be deleted or rotation, Apache will not restart. A restart of Apache is required to prevent Apache from continuing to write the files cpanellogd will process. Without the restart there is potential for data loss. See "File Descriptors" for a detailed explanation. When restarting Apache, a graceful restart is requested. This type of restart typically places the least load on a server and should maintain service up-time 2. After processing is complete, respective statistical information is available from the normal interfaces: Daily Process Log, View Bandwidth Usage interfaces, or various domain statistic portals (i.e., Webalizer, AWStats, Analog).

Report Discrepancies
Any discrepancy between statistical reports will stem from:
2

See Section 6.1, Apache Restarts for known issues with Apache restarts.

WHM/cPanel Log Processing

1. Timing between log/bandwidth parsings 2. Inherent differences of the report engines and the data they report File Descriptors When an application opens a file the application receives a file descriptor( hereafter FD) from the operating system. This FD is an index into a kernel-resident data structure containing the details of all open files. The common system utility lsof allows the admin to view the list of all open files and which applications are using them. The FD remains associated with the requested file as long as the application keeps the file open. Changes to the underlying file have little impact on the FD or the holding application. The application only knows about the FD. When performing certain operations on files that are opened by long running services it is often necessary to restart the services. This prevents the situations shown in Example 2, Renaming a file with an open FD and Example 3, Deleting a file with an open FD.

Example 2. Renaming a file with an open FD

1. Application Foo opens /etc/named.conf for reading and writing. 2. Foo receives a FD associated with the file. 3. The system admin renames /etc/named.conf to /etc/named.conf.bak 4. Foo writes new data to the FD obtained in step 1 5. The new data is added to /etc/named.conf.bak

Example 3. Deleting a file with an open FD

1. Application Foo opens /etc/named.conf for reading and writing. 2. Foo receives a FD associated with the file. 3. The system admin deletes /etc/named.conf 4. Foo writes new data to the FD obtained in step 1 5. The new data disappears as the FD no longer points to a valid file.

3.2. Rotation & Retention

Log files can be selected in the respective interface for rotation inclusion.(See Section 4.2, Log Rotation). If a given cPanel or Apache log file is rotated, it will occur once the file exceeds 300MB (See Section 4.2, Log Rotation for specifying a custom value). File limits are checked during each parse cycle.

WHM/cPanel Log Processing

Domain access Apache logs do not rotate by default. However, the logic and configuration that governs the retention of this data is separate of cPanel and Apache log files. The most obvious benefit of this approach is domain users can access and manage these raw logs. The threshold for rotation and file trimming on domain logs is set at 1GB3. Domain users have the option to archive domain logs on a monthly basis. Additionally, they have the ability to automatically remove the previous months archive. The WHM/cPanel administrator has the ability retain domain logs on a monthly basis (See Domain Log Rotation). Additionally, they can elect to delete domain logs after parsing completes.

Note
Deletion is negated if the administrator has also elected to keep logs each month or if the affected domain user has elected to archive their files.

3.3. Synchronicity
Log parsers are not created equal. It is by design that different log parsers expend varying levels of system resources determining similar, if not identical, information. Additionally, each domain experiences different traffic, generating various length logs that must be parsed. Depending on the number of accounts and the traffic they receive, it is possible, and at some point highly probable on certain server setups, that parsing will not complete for all accounts within a parsing cycle. Processing domain logs is the slowest part of the log parsing process. The log parsers provided by cPanel include: Analog AWStats Webalizer Only the parsers enabled in WHM are active during a log processing period. The parsers are executed serially, rather than in parallel, to reduce system load the parsers generate.

3.3.1. Simple Example

Server "blue.example.com" has 5 accounts: 1. example.com 2. ejemplo.info 3. beispiel.com 4. voorbeeld.net 5. esempio.org The majority of "blue.example.com"'s traffic is on domain esempio.org. The first time cpanellogd processes the domains, it does so in the following order: 1. beispiel.com
3

As of 11.25.0, this threshold is fixed and cannot be configured to a different value.

WHM/cPanel Log Processing

2. ejemplo.info 3. esempio.org 4. example.com 5. voorbeeld.net The first two domains are processed very quickly. Because esempio.org has more data, it takes many minutes to process. When the appropriate entry is made in /var/cpanel/lastrun the timestamps for example.com and voorbeeld.net are 'not in sync' with the first three domains because the gap between the earliest and latest time stamp will be greater than 500 seconds. Once the Log Processing Frequency threshold is exceeded, beispiel.com, ejemplo.info, esempio.org will be processed but not the other two domains since they were completed sometime just within the threshold. At cpanellogd's next domain log parse cycle, cpanellogd will now process the remaining domains. Due to the above, there are now 2 Apache restarts, 15 minutes apart4.

4. Configuration
4.1. Cycle Frequency
Log Process Frequency
Interface Description Affected Files and Directories WHM >> Main >> Server Configuration >> Statistic Software Configuration >> Statistics Configuration Minimum time threshold before attempting to parse Log files. /usr/local/apache/logs/ /home/${domainowner}/tmp/analog/ /home/${domainowner}/tmp/awstats/ /home/${domainowner}/tmp/webalizer/ /home/${domainowner}/tmp/webalizerftp/ 24 Hours

Default Value Units

Bandwidth Process Frequency

Interface Description Affected Files and Directories WHM >> Main >> Server Configuration >> Statistic Software Configuration >> Statistics Configuration Minimum time threshold before attempting to parse Bandwidth files. /usr/local/cpanel/logs/ /var/cpanel/bandwidth/ /home/${domainowner}/tmp/cpbandwidth/ /home/${domainowner}/tmp/bw-${domainowner}-7days.png /home/${domainowner}/tmp/bw-${domainowner}-today.png

Approximate. cpanellogd makes a processing pass every 500 seconds. Each pass alternates between domain traffic logs and bandwidth monitoring logs. This simple example excludes any other processes influencing Apache.

WHM/cPanel Log Processing

/home/${domainowner}/tmp/bw-${domainowner}-year.png Default Value Units 2 Hours

Black-out Hours
Interface Description WHM >> Main >> Server Configuration >> Statistic Software Configuration >> Statistics Configuration >> Statistics Schedule Specific hours to suspend cpanellogd operation. Any active parse cycle that lapses into a black-out time slot will complete for the current domain being processed; all remaining domains queued are left unprocessed until the next parse cycle. Processing is suspended and black-out is honored . Once black-out period ends cpanellogd commences with a normal parse cycle.

Affected Files and Directories Default Value Units No black-outs by default Hour long blocks via form checkbox.

Deference to Backup
Interface Description WHM >> Main >> Server Configuration >> Statistic Software Configuration >> Statistics Configuration >> Statistics Schedule If enabled cpanellogd will be suspended while backups are run. Any active parse cycle will complete for the current domain before deference to backup is honored. Any remaining domains queued will be processed at the next parse cycle proceeding backups. This behavior is identical to Black-out Hours. /var/cpanel/cpanel.config Disabled Boolean via form checkbox.

Affected Files and Directories Default Value Units

Emulating Apache's Prior Behavior

Description Emulating Apache and log parsing behavior prior to v11.25.0 is not directly supported. However, approximate behavior can be accomplished by configuring Black-out Hours or modifying /var/cpanel/lastrun/* timestamps5.

4.2. Log Rotation

Apache Log Rotation
Interface Description
5

WHM >> Main >> Server Configuration >> Apache Configuration >> Log Rotation Select which Apache log files to include in rotation.

Modifying sub-system upkeep files is discouraged.

WHM/cPanel Log Processing

Affected Files and Directories Default Value Units

/var/cpanel/cpanel.config /var/cpanel/log_rotation/ Disabled Boolean via form checkbox.

Domain Log Rotation

Interface Description WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs Elect to rotate domain specific Apache log files by 'checking' "Keep log files at the end of the month."

Important
Domain logs are hard coded to rotate at 1GB, as well as at the end of each month. Elect to retain data within domain Apache logs by 'unchecking' "Delete each domain's access log after stats run." Affected Files and Directories Default Value Units /var/cpanel/cpanel.config /usr/local/apache/domlogs/ Disabled Boolean via form checkbox.

cPanel Log Rotation

Interface Description Affected Files and Directories Possible Side Effects Default Value Units WHM >> Main >> Server Configuration >> cPanel Log Rotation Configuration Select which cPanel log files to include in rotation. /var/cpanel/cpanel.config /var/cpanel/log_rotation/ See Threshold for Log Rotation no files are included in rotation Boolean via form checkbox.

Threshold for Log Rotation

Interface Description /var/cpanel/cpanel.config Defining variable "rotatelogs_size_threshhold_in_megabytes" within cpanel.config will determine how large cPanel or Apache logs files are allowed to grow before rotation occurs.

Warning
This value applies to both Apache and cPanel log files but has no affect of Domain access Apache logs

WHM/cPanel Log Processing

Note
This variable is excluded in the initial configuration of cpanel.config and therefore must be appended manually. Additionally any value <= 10 will be ignored. Affected Files and Directories /var/cpanel/cpanel.config /usr/local/apache/logs/ /usr/local/cpanel/logs/ 300 Megabytes

Default Value Units

4.3. WHM Tweak Settings

Log Processing Frequency
Interface Description Status WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs See Log Process Frequency Invalid exposure of functionality. Schedule for deprecation 11.25.X.

Bandwidth Processing Frequency

Interface Description Status WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs See Bandwidth Process Frequency Invalid exposure of functionality. Schedule for deprecation 11.25.X.

Keep Stats Log

Interface Description WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs Elect to not purge cPanel's statistic log upon cPanel restart.

Note
cPanel log rotation also affects this file. Affected Files and Directories Default Value Units /usr/local/cpanel/logs/stats_log Disabled Boolean via form checkbox.

Keep Logs
Interface Description WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs Elect to retain domain access logs, rotating them at the end of the month.

10

WHM/cPanel Log Processing

Note
Rotation and retention policy is enacted anytime these logs exceed 1GB. See Domain Log Rotation.

Warning
Election to retain these logs can quickly fill disk space! Affected Files and Directories Default Value Units Disabled Boolean via form checkbox.

Purge Logs
Interface Description WHM >> Main >> Server Configuration >> Tweak Settings >> Stats and Logs Delete domain access logs after parsing occurs.

Note
Electing to Keep Logs will override this configuration. See Keep log files at the end of the month.

Note
If a domain user electing to Archive Logs, it will override this configuration for their affected files. See Archive Logs in your home directory at the end of each month.

Warning
Election to retain these logs can quickly fill disk space! Affected Files and Directories Default Value Units Enabled Boolean via form checkbox.

4.4. cPanel Options

Archive Logs
Interface Description cPanel >> Logs >> Raw Access Logs >> Configure Logs Archive domain access logs on a monthly basis.

Note
Purge Logs will be negated for domain user if this configuration is enabled. See Delete each domain's access logs after stats run.

11

WHM/cPanel Log Processing

Affected Files and Directories Default Value Units Disabled Boolean via form checkbox.

Purge Archive
Interface Description cPanel >> Logs >> Raw Access Logs >> Configure Logs Delete previous months archive on a monthly basis.

Warning
Electing to NOT Purge Archive can consume excessive disk space and disk quota. See Delete each domain's access logs after stats run. Affected Files and Directories Default Value Units Disabled Boolean via form checkbox.

4.5. Flag Files

Interface Description Flag File Affected Files and Directories Default Value Units Nonexistent Zero length file. CLI Zero length file that disables Bandwidth monitoring. /etc/rrdtooldisable

4.6. Other Logs

N/A

5. Troubleshooting
5.1. Unprocessed .bkup files? In the extreme event that a parser does not properly parse it's data, the result will be two-fold: 1. cpanellogd will attempt to reprocess ${logfile}.bkup. 2. If one is trying to view the report while the parser is working, a report should be rendered based on the older, finished report, with reference to unparsed or incomplete data 6.
6

5.2.

Why are all domains not processed during a specific cycle?

12

WHM/cPanel Log Processing

Domain logs are parsed end to end. If processing for one domain takes several minutes, it is likely that the following domains will not complete their log parsing within the same 500 second parsing cycle; ergo, the latter domains will receive future parsing in a cycle time different than the former domains.

6. Known Issues and Caveats

6.1. Apache Restarts
6.1.1. Apache 2.2 and Child Processes in 'G' Status
Reference Errata Status Bug Apache 2.2 Child Processes [http://docs.cpanel.net/twiki/bin/vief/EasyApache3/EA3KnownIssues#Bug_Apache_2_2_Child_Processes_i] Apache Foundation considers this issue resolved as of Apache v2.2.12+ Resolved

6.1.2. Apache and Child Processes in 'K' Status

Behavior Possible Cause Possible Fix Status If Apache has several child processes in 'K' (KeepAlive) status, a graceful restart can take several minutes, during which Apache will not accept any new connections. KeepAlive directives in httpd.conf Limit or exclude KeepAlive directives in httpd.conf. Unconfirmed

7. Conclusion
WHM/cPanel 11.25.0 provides new mechanisms for maximizing server control and better resource utilization. cpanellogd, the daemon responsible for parsing various logs and triggering report engines, has changed internally and how it negotiates with server processes; specifically, Apache is managed for scalability and performance. New configurations can be managed in WHM.

Glossary
Glossary of terms used in this document and within the WHM/cPanel produce.

B
Bandwidth Bandwidth Logs The measure of data that traverses a network interface. See Also Bandwidth Logs. Log files detailing all traffic on a per domain basis. This includes all HTTP (without headers), HTTPS (with headers), mail and FTP transmissions.

D
Domain Access Apache Logs Access logs generated by Apache specific to a domain.

13

WHM/cPanel Log Processing

Domain Logs

See Domain Access Apache Logs.

L
Logs See Domain Access Apache Logs.

14