Apache Crashing and Restarting

[aaronjwood]

Hello,

I am running the newest version of OTRS on a Windows Server 2003 R2. I am getting this error in my Windows Application event log:

Faulting application httpd.exe, version 2.2.17.0, faulting module perl512.dll, version 0.0.0.0, fault address 0x000206e3.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Here is my apache log after clearing it and getting first error.

Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 10:16:00 2012] [notice] Apache/2.2.17 (Win32) mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.12.3 configured -- resuming normal operations
[Fri Apr 06 10:16:00 2012] [notice] Server built: Oct 18 2010 01:58:12
[Fri Apr 06 10:16:00 2012] [notice] Parent: Created child process 3744
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 10:16:06 2012] [notice] Child 3744: Child process is running
[Fri Apr 06 10:16:06 2012] [notice] Child 3744: Acquired the start mutex.
[Fri Apr 06 10:16:06 2012] [notice] Child 3744: Starting 64 worker threads.
[Fri Apr 06 10:16:06 2012] [notice] Child 3744: Starting thread to listen on port 80.
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 10:19:46 2012] [warn] [client 10.0.1.251] incomplete redirection target of '/otrs/index.pl' for URI '/' modified to 'http://equotes.suntire.com/otrs/index.pl'
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 10:43:35 2012] [notice] Parent: child process exited with status 3221225477 -- Restarting.
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 10:43:39 2012] [notice] Apache/2.2.17 (Win32) mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.12.3 configured -- resuming normal operations
[Fri Apr 06 10:43:39 2012] [notice] Server built: Oct 18 2010 01:58:12
[Fri Apr 06 10:43:39 2012] [notice] Parent: Created child process 3480
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 10:43:45 2012] [notice] Child 3480: Child process is running
[Fri Apr 06 10:43:45 2012] [notice] Child 3480: Acquired the start mutex.
[Fri Apr 06 10:43:45 2012] [notice] Child 3480: Starting 64 worker threads.
[Fri Apr 06 10:43:45 2012] [notice] Child 3480: Starting thread to listen on port 80.
[Fri Apr 06 10:43:45 2012] [error] [client 10.0.1.251] File does not exist: C:/Program Files/OTRS/Apache/htdocs/favicon.ico
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1

Any ideas about what is going on here???

[jojo]

any more insights from the OTRS log and the apache error log?

Whats the exact version you installed?

[aaronjwood]

I am running version 3.1.2. Another crash/restart happened at 14:27:35 on my server's time. Reporting below.

Here is the error from Windows event viewer.

=== BEGIN WINDOWS ERROR ===

Faulting application httpd.exe, version 2.2.17.0, faulting module perl512.dll, version 0.0.0.0, fault address 0x0001c293.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


=== END WINDOWS ERROR ===

Here is the portion of the otrs.log file for that time frame

=== BEGIN OTRS LOG ===

[Fri Apr 6 13:46:38 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'minorcy@aetna.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000235] Sun Tire E-Quote Request;
[Fri Apr 6 13:48:53 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'lizardlassie@aol.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000226] Sun Tire E-Quote Request;
[Fri Apr 6 13:50:06 2012][Notice][Kernel::System::PostMaster::FollowUp::Run] FollowUp Article to Ticket [2012040610000244] created (TicketID=509, ArticleID=1385). ,
[Fri Apr 6 13:50:06 2012][Notice][Kernel::System::MailAccount::IMAPS::_Fetch] IMAPS: Fetched 1 email(s) from equotes@suntireinc.com/imap.googlemail.com.
[Fri Apr 6 13:57:11 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'lizardlassie@aol.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000217] Sun Tire E-Quote Request;
[Fri Apr 6 13:59:16 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'PJUBIN@its.jnj.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000244] Sun Tire E-Quote Request;
[Fri Apr 6 14:18:43 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'raygann@gmail.net,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000182] Sun Tire E-Quote Request;
[Fri Apr 6 14:20:07 2012][Notice][Kernel::System::Ticket::TicketCreate] New Ticket [2012040610000253/Delivery Status] created (TicketID=510,Queue=Sun Tire E-Quotes,Priority=3 normal,State=new)
[Fri Apr 6 14:20:08 2012][Notice][Kernel::System::Ticket::Article::SendAutoResponse] Sent no auto response to 'Mail Delivery Subsystem <mailer-daemon@googlemail.com>' because config option SendNoAutoResponseRegExp (/(MAILER-DAEMON|postmaster|abuse)@.+?\..+?/i) matched.
[Fri Apr 6 14:20:08 2012][Notice][Kernel::System::MailAccount::IMAPS::_Fetch] IMAPS: Fetched 1 email(s) from equotes@suntireinc.com/imap.googlemail.com.
[Fri Apr 6 14:29:22 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'rossparrish3@aol.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000208] Sun Tire E-Quote Request;
[Fri Apr 6 14:34:40 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'vaknudsen@hotmail.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000191] Sun Tire E-Quote Request;
[Fri Apr 6 14:38:59 2012][Notice][Kernel::System::Ticket::Article::ArticleSend] Sent email to 'raygann@gmail.com,' from 'Sun Tire E-Quote System <equotes@suntireinc.com>'. HistoryType => SendAnswer, Subject => Re: [Ticket#2012040610000253] Delivery Status Notification (Failure);

=== END OTRS.LOG ===

Here is the portion of the apache error log for that time frame.

=== BEGIN APACHE ERROR LOG ===

Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 14:27:37 2012] [notice] Parent: child process exited with status 3221225477 -- Restarting.
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 14:27:43 2012] [notice] Apache/2.2.17 (Win32) mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.12.3 configured -- resuming normal operations
[Fri Apr 06 14:27:43 2012] [notice] Server built: Oct 18 2010 01:58:12
[Fri Apr 06 14:27:43 2012] [notice] Parent: Created child process 2024
Scalars leaked: 1
Scalars leaked: 1
Scalars leaked: 1
[Fri Apr 06 14:27:49 2012] [notice] Child 2024: Child process is running
[Fri Apr 06 14:27:49 2012] [notice] Child 2024: Acquired the start mutex.
[Fri Apr 06 14:27:49 2012] [notice] Child 2024: Starting 64 worker threads.
[Fri Apr 06 14:27:49 2012] [notice] Child 2024: Starting thread to listen on port 80.
Scalars leaked: 1
Scalars leaked: 1

=== END APACHE ERROR LOG ===

[aaronjwood]

I think I have determined that Apache is crashing when I log into the server via a RDP connection. Any ideas why this would be happening?

[jojo]

not enough memory?

[aaronjwood]

OK, so I got frustrated with Apache's random crashes on the Windows 2003 Server box so I used and old computer that we had laying around and installed Ubuntu 11 on it. I did a fresh install of OTRS 3.1.2 as described at http://wiki.otterhub.org/index.php?titl ... %2810.4%29. I then exported my mysql database from the windows box and imported it to the linux box. Everything went very smooth, but I am now having the same exact issue on this box. Apache will randomly crash with a segmentation fault. This computer has nothing else loaded on it and is 100% dedicated to OTRS. It seems that it happens when an agent is submitting a response to a ticket. When they click to send the response, Apache crashes and they receive a zero length response error from squid. The logs provide no details on why this is happening. Is there anything else I can do to try and track the error down? Any help is greatly appreciated.

[aaronjwood]

I enabled core dumps and then analyzed the dump with gdb and got the below. Any ideas what is going on here.

root@ubuntu11:~# cd /tmp/apache2-gdb-dump/
root@ubuntu11:/tmp/apache2-gdb-dump# gdb -c core
GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>.
[New LWP 21635]
[New LWP 21638]
[New LWP 21640]
[New LWP 21641]
[New LWP 21642]
[New LWP 21643]
[New LWP 21644]
[New LWP 21645]
[New LWP 21646]
[New LWP 21647]
[New LWP 21648]
[New LWP 21649]
[New LWP 21650]
[New LWP 21651]
[New LWP 21652]
[New LWP 21653]
[New LWP 21654]
[New LWP 21655]
[New LWP 21656]
[New LWP 21657]
[New LWP 21658]
[New LWP 21659]
[New LWP 21660]
[New LWP 21661]
[New LWP 21662]
[New LWP 21663]
[New LWP 21639]
Core was generated by `/opt/otrs/bin/cgi -k start'.
Program terminated with signal 11, Segmentation fault.
#0 0x00622416 in __kernel_vsyscall ()
(gdb) backtrace full
#0 0x00622416 in __kernel_vsyscall ()
No symbol table info available.
#1 0x00ae6a8b in ?? ()
No symbol table info available.
#2 0x00ab32f8 in ?? ()
No symbol table info available.
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[aaronjwood]

Well, when Apache was crashing on the windows machine the event log showed that the perl dll was the cause. I have disabled mod_perl on the Ubuntu machine and am currently running otrs using only cgi. Although it is slower, it seems to be stable. Since making the modification there has not been a single segfault in the Apache logs. I suppose there is a some issue with mod_perl/ubuntu??? I don't know...any suggestions?

[kishore]

I have been facing the similar problem on windows as reported here
viewtopic.php?f=81&t=12957

For me it happens when ever some agent tries to close the ticket.
But did not find a solution yet.

[Mike_B]

Hi all,

I've created a new version of the Windows Installer which bundles a different
Apache (ApacheLounge's version) with a different mod_perl; for me it no longer
restarts but I would like to get some feedback from actual users.

Please test it and leave feedback in this bug report:

https://www.dropbox.com/s/3zwhmjc6tx2td ... -beta3.exe
http://bugs.otrs.org/show_bug.cgi?id=6399