Archive for the ‘Apache’ Category.
Well thought I would drop in and writeup a quick fix to a minor annoyance with the IUS (and other 5.4 versions of php) and that is the STRICT and Deprecated warnings in logs. It appears that the new version of this is a bit more “open” on what it takes to turn this on (ie code in sites like phpbb, wordpress, and others) and the error_reporting setting in php.ini didnt make it quiet. Lucky for me this quick hack did:
LoadModule php5_module modules/libphp5.so
php_admin_value error_reporting ” E_ALL & ~E_DEPRECATED & ~E_STRICT”
Global override saves the day. Now I can go back to sed-ing out all this cruft this update to 5.4IUS on RHEL 6 just did to my logs! I will say that the mysqlnd that comes with 5.4 is quite impressive…look for a writeup sometime soon.
Was troubleshooting a coding issue for a customer the other day when I noticed the number of requests currently being processed. I am going to write about this in the coming days. Basically there was a code issue (we do not support code for this customer) that caused a large number of “sleep” states in MySQL as well as a huge number of ‘
W Sending Reply’ states in Apache.
- Current Time: Thursday, 11-Feb-2010 11:47:54 CST
- Restart Time: Thursday, 11-Feb-2010 09:17:12 CST
- Parent Server Generation: 0
- Server uptime: 2 hours 30 minutes 41 seconds
- Total accesses: 226733 – Total Traffic: 10.8 GB
- CPU Usage: u1812.56 s80.08 cu.04 cs0 – 20.9% CPU load
- 25.1 requests/sec – 1.2 MB/second – 49.8 kB/request
- 1337 requests currently being processed, 113 idle workers
OS: CentOS 4.8
Apache : Custom RPM from source with only a single change to the location of the suexec directory
strace -t -f -v -p $process -o /path/to/outputfile (note the $process is the primary Apache Process)
To find the primary Apache Process you do a :
ps -ef | grep httpd
and it returns something like this :
apache 26898 22378 8 13:50 ? 00:00:01 /usr/sbin/httpd -k start
the second number 22378 is the PID of the Apache parent process. I then waited for a :
Dec 11 10:02:20 web02 kernel: httpd: segfault at 0000007fbf3fff0c rip 0000002a9567344a rsp 0000007fbf3ffe90 error 6
in my /var/log/messages. Once that came I did a:
grep SIGSEGV /path/to/file_generated_w/strace
and noted times and PIDs. Here is a example output :
19730 12:07:35 — SIGSEGV (Segmentation fault) @ 0 (0) —
19784 12:08:56 — SIGSEGV (Segmentation fault) @ 0 (0) —
I then grepped out the PIDs (19784 and 19730 in the above example) with a segfault to different files and began reading. To grep this out I did :
grep 19730 /path/to/file_generated_w/strace > /tmp/out.19730
It was in these files I found my problem. Your mileage may vary but I found this method much easier than using the Apache config setting of CoreDumpDirectory which requires several changes that have to be undone. The CoreDumpDirectory setting also requires a few restarts of Apache which in a production environment can be undesired.
The main caveat to using strace is that , on a busy server, you can generate 100-300M of logs per minute so make sure you have the diskspace on the partition you are sending out strace output.