ERRORS "Use of uninitialized value"/"Attempt to free"

[dolfiz]

Hi all,

we've got too many errors on log and we can't explain the reason behind them. Most of all are of this kind:

Code: Select all

[Mon Mar 31 12:16:31 2014] -e: Use of uninitialized value in hash element at /opt/otrs//Kernel/Modules/AgentTicketProcess.pm line 1426.
[Thu Apr  3 09:48:26 2014] -e: Use of uninitialized value in string ne at /opt/otrs//Kernel/Output/HTML/LayoutTicket.pm line 735.
[Thu Apr  3 10:25:38 2014] -e: Use of uninitialized value $Param{"Value2"} in string eq at /opt/otrs//Kernel/System/DynamicField/Driver/Base.pm line 33.
[Thu Apr  3 10:52:47 2014] -e: Use of uninitialized value in string ne at /opt/otrs//Kernel/Output/HTML/LayoutTicket.pm line 735.
[Thu Apr  3 10:58:49 2014] -e: Use of uninitialized value $KeyValue[1] in concatenation (.) or string at /opt/otrs//Kernel/Output/HTML/TicketOverviewMenuSort.pm line 123.

or, alternatively, of this other kind:

Code: Select all

[Mon Mar 31 11:57:21 2014] -e: Attempt to free unreferenced scalar: SV 0x7fbc9f79b338, Perl interpreter: 0x7fbca1935230 at /opt/otrs//Kernel/System/DB.pm line 639.
[Mon Mar 31 13:24:07 2014] -e: Attempt to free unreferenced scalar: SV 0x7fbce0c3a5e8, Perl interpreter: 0x7fbd0f00f560 at /opt/otrs//Kernel/System/Cache/FileStorable.pm line 116.
[Tue Apr  1 09:49:12 2014] -e: Attempt to free unreferenced scalar: SV 0x7fbc856d80e0, Perl interpreter: 0x7fbcc5d109e0 at /opt/otrs//Kernel/System/GenericAgent.pm line 955.
[Tue Apr  1 12:45:10 2014] -e: Attempt to free unreferenced scalar: SV 0x7fbc99f28ec0, Perl interpreter: 0x7fbcb8a88360 at /opt/otrs/bin/cgi-bin/index.pl line 41.
[Wed Apr  2 10:11:58 2014] -e: Attempt to free unreferenced scalar: SV 0x7fbca7788898, Perl interpreter: 0x7fbcb9b13cb0 at /opt/otrs/Kernel/cpan-lib/Mail/Address.pm line 58.

Sometimes, after these errors, apache throw a "glibc error" and restart itself:

Code: Select all

[Thu Apr  3 11:04:10 2014] -e: Use of uninitialized value in -e at /opt/otrs//Kernel/System/Main.pm line 353.
[Thu Apr  3 11:04:17 2014] -e: Attempt to free unreferenced scalar: SV 0x7fbcc87db6b8, Perl interpreter: 0x7fbcd0e227d0 at /opt/otrs//Kernel/System/GenericAgent.pm line 275.
[Thu Apr 03 11:04:35 2014] [error] :Apache2 IO flush: (103) Software caused connection abort at /opt/otrs//Kernel/Output/HTML/Layout.pm line 1709
:Apache2 IO flush: (103) Software caused connection abort at -e line 0:Apache2 IO flush: (103) Software caused connection abort at -e line 0Scalars leaked: 1
Scalars leaked: -2
[Thu Apr  3 11:05:22 2014] -e: Use of uninitialized value $KeyValue[1] in concatenation (.) or string at /opt/otrs//Kernel/Output/HTML/TicketOverviewMenuSort.pm line 123.
Scalars leaked: 2
[Thu Apr  3 11:05:46 2014] -e: Use of uninitialized value $Param{"To"} in string ne at /opt/otrs//Kernel/Modules/AgentTicketCompose.pm line 1757.
*** glibc detected *** /usr/sbin/apache2: munmap_chunk(): invalid pointer: 0x00007fbcdc262e50 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fbd0c473b96]
/usr/lib/libperl.so.5.14(Perl_sv_clear+0x367)[0x7fbd09418987]
/usr/lib/libperl.so.5.14(Perl_sv_free2+0x52)[0x7fbd094192a2]
/usr/lib/libperl.so.5.14(+0xb6c61)[0x7fbd09413c61]
/usr/lib/libperl.so.5.14(perl_destruct+0x1032)[0x7fbd093a6ef2]
/usr/lib/apache2/modules/mod_perl.so(modperl_perl_destruct+0x5f)[0x7fbd096fb81f]
[...]

Do you have suggestion to properly investigate the reasons behind these problems? The CPU load seems to be high, but we experienced those errors also on low load situations.

Any help is appreciated.
Thanks,
Luca

P.s.: PERL version v5.14.2

[mathiask]

Have you checked your path? Looks like you got a "/" to much here:

/opt/otrs//Kernel/

[reneeb]

Perl can handle that...

Which versions of Apache, mod_perl and OTRS do you use? I have several development environments with OTRS and Perl 5.14.2 under Linux without any problems...

[dolfiz]

Perl can handle that...

Do you mean that we can ignore those kind of errors?

Which versions of Apache, mod_perl and OTRS do you use? I have several development environments with OTRS and Perl 5.14.2 under Linux without any problems...

Code: Select all

root@itssg-vlotrs01:~# perl -v

This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi
(with 56 registered patches, see perl -V for more detail)

root@itssg-vlotrs01:~# perl -Mmod_perl2\ 999
mod_perl2 version 999 required--this is only version 2.000005.
BEGIN failed--compilation aborted.

root@itssg-vlotrs01:~# apache2ctl -v
Server version: Apache/2.2.22 (Ubuntu)
Server built:   Jul 12 2013 13:37:15

OTRS v3.3.3

I don't know if the following can be of any help:

Code: Select all

  o Crypt::Eksblowfish::Bcrypt.......ok (v0.008)
  o Crypt::SSLeay....................ok (v0.57)
  o Date::Format.....................ok (v2.24)
  o DBI..............................ok (v1.616)
  o DBD::mysql.......................ok (v4.020)
  o DBD::ODBC........................Not installed! (optional - Required to connect to a MS-SQL database.)
  o DBD::Oracle......................Not installed! (optional - Required to connect to a Oracle database.)
  o DBD::Pg..........................Not installed! (optional - Required to connect to a PostgreSQL database.)
  o Encode::HanExtra.................ok (v0.23)
  o GD...............................ok (v2.46)
    o GD::Text.......................ok (v0.86)
    o GD::Graph......................ok (v1.44)
  o IO::Socket::SSL..................ok (v1.53)
  o JSON::XS.........................ok (v2.32)
  o LWP::UserAgent...................ok (v6.05)
  o Mail::IMAPClient.................ok (v3.30)
    o IO::Socket::SSL................ok (v1.53)
  o ModPerl::Util....................ok (v2.000005)
  o Net::DNS.........................ok (v0.66)
  o Net::LDAP........................ok (v0.43)
  o Net::SSL.........................ok (v2.84)
  o PDF::API2........................ok (v2.019)
    o Compress::Zlib.................ok (v2.033)
  o Text::CSV_XS.....................ok (v0.85)
  o Time::HiRes......................ok (v1.9721_01)
  o XML::Parser......................ok (v2.41)
  o YAML::XS.........................ok (v0.38)

Bundled modules:

  o Algorithm::Diff..................ok (v1.1902)
  o Apache2::Reload..................ok (v0.12)
  o Apache::DBI......................ok (v1.12)
  o Authen::SASL.....................ok (v2.16)
  o CGI..............................ok (v3.63)
  o CSS::Minifier....................ok (v0.01)
  o Class::Inspector.................ok (v1.28)
  o Crypt::PasswdMD5.................ok (v1.3)
  o Date::Pcalc......................ok (v1.2)
  o Email::Valid.....................ok (v0.190)
  o Encode::Locale...................ok (v1.03)
  o IO::Interactive..................ok (v0.0.6)
  o JSON.............................ok (v2.59)
  o JSON::PP.........................ok (v2.27202)
  o JavaScript::Minifier.............ok (v1.05)
  o LWP..............................ok (v6.05)
  o Linux::Distribution..............ok (v0.21)
  o Locale::Codes....................ok (v3.26)
  o MIME::Tools......................ok (v5.504)
  o Mail::Address....................ok (v2.12)
  o Mail::Internet...................ok (v2.12)
  o Module::Refresh..................ok (v0.17)
  o Mozilla::CA......................ok (v20130114)
  o Net::HTTP........................ok (v6.06)
  o Net::IMAP::Simple................ok (v1.2201)
  o Net::SSLGlue.....................ok (v1.04)
  o Proc::Daemon.....................ok (v0.14)
  o SOAP::Lite.......................ok (v1.06)
  o Sys::Hostname::Long..............ok (v1.4)
  o Text::CSV........................ok (v1.32)
  o Text::Diff.......................ok (v1.41)
  o URI..............................ok (v1.60)
  o YAML.............................ok (v0.84)
  o parent...........................ok (v0.227)

[dolfiz]

Have you checked your path? Looks like you got a "/" to much here:

/opt/otrs//Kernel/

I've seen it but I don't think that it's a problem. If that was a problem it should not have been able to execute any of those .pm files, but it clearly executed them.

[reneeb]

Perl can handle that...

Do you mean that we can ignore those kind of errors?

Sorry, I should have quoted matiask' answer. Perl can handle the doubled slashes in the file paths.

Which versions of Apache, mod_perl and OTRS do you use? I have several development environments with OTRS and Perl 5.14.2 under Linux without any problems...

Code: Select all

root@itssg-vlotrs01:~# perl -v

This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi
(with 56 registered patches, see perl -V for more detail)

root@itssg-vlotrs01:~# perl -Mmod_perl2\ 999
mod_perl2 version 999 required--this is only version 2.000005.
BEGIN failed--compilation aborted.

root@itssg-vlotrs01:~# apache2ctl -v
Server version: Apache/2.2.22 (Ubuntu)
Server built:   Jul 12 2013 13:37:15

OTRS v3.3.3

OK. I'll have a look at the combinations I run...

[dolfiz]

OK. I'll have a look at the combinations I run...

Thank you very much, we really appreciate it.

[reneeb]

So far, I can see only two differences:

1) I have no OTRS 3.3.3 in my dev boxes. I have some 3.0.x, some 3.2.x and 3.3.0 beta 5, 3.3.5, 3.3.6. So this shouldn't be an issue
2) Most of my dev boxes run on a 32-bit-Linux. The 64-bit-boxes aren't heavily used yet. If I have enough time over the weekend I will setup a performance test for one 64-bit box to get some load on the box...

[dolfiz]

If I have enough time over the weekend I will setup a performance test for one 64-bit box to get some load on the box...

Thank you so much.

I don't know if this can be of some interest, but the system began to show those kind of errors especially after we get some heavy load on the system. They were present before, but not so often. Currently the environment is composed by two virtual machines:

Code: Select all

WEB SERVER:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Stepping:              5
CPU MHz:               1995.000
BogoMIPS:              3990.00
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-7

DB SERVER:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Stepping:              5
CPU MHz:               1995.000
BogoMIPS:              3990.00
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-7

And this is the volumes that they handle:
Agents 143
Articles 164826
Articles per ticket (avg) 2.00
Customers with at least one ticket 31959
Groups 18
Install directory /opt/otrs
Months between first and last ticket 1.97
Operating system linux
Product OTRS 3.3.3
Roles 18
Tickets 82454
Tickets per month (avg) 41788.73

With this heavy load, we've been forced to make a workaround using generic agent in order to differentiate tickets at every queue update. Now there are so many generic agent of this kind (at least 100) and I think that this is obviously not the way they are meant to be used. We are refactoring all the configuration, in order to make it more rationale and simple, but we are waiting some clarification from our clients. In the meanwhile we need to keep on with this workaround. Does someone has experience of this kind? Can too many generic agents fired at the same time be a problem for OTRS? I'm thinking about some cuncurrency issue. With hundreds of ticket per-hour, with so many queue update, I thought that this can be a problem.

Thanks again,
Luca

[reneeb]

I don't know if this can be of some interest, but the system began to show those kind of errors especially after we get some heavy load on the system. They were present before, but not so often.

This is of interest. I guess with high load mod_perl get hicksup that leads to the error.

With this heavy load, we've been forced to make a workaround using generic agent in order to differentiate tickets at every queue update. Now there are so many generic agent of this kind (at least 100) and I think that this is obviously not the way they are meant to be used. We are refactoring all the configuration, in order to make it more rationale and simple, but we are waiting some clarification from our clients. In the meanwhile we need to keep on with this workaround. Does someone has experience of this kind? Can too many generic agents fired at the same time be a problem for OTRS? I'm thinking about some cuncurrency issue. With hundreds of ticket per-hour, with so many queue update, I thought that this can be a problem.

Too many GenericAgents running at the same time might cost performance of the system. Theoretically there shouldn't be race conditions (but I have no systems with such a heavy load)...

[dolfiz]

This is of interest. I guess with high load mod_perl get hicksup that leads to the error.

Too many GenericAgents running at the same time might cost performance of the system. Theoretically there shouldn't be race conditions (but I have no systems with such a heavy load)...

I guessed so. I'm pushing everyone in order to speed up the "model" refactoring. We're going to delete those generic agents and to reduce number of queues an states.

By the way, there is another complexity factor represented by the process (build using process management) that manages the ticket workflow. It is complex with dozen of transitions (we almost reached the maximum complexity possible, because the javascript app clearly becomes so so heavy and slow). I think that this functionality is not mean to be used in this way too. I suggest everyone out there to use it only to map simple processes. With hundreds of tickets, adding many transitions causes tickets to miss actions sometimes (especially lock and state changes).

Hope these remarks can be of any help for others in the same situation of us, or for OTRS devs in order to improve this great tool.

Thank you,
Luca