OTRS slowness and performance improvement.

[harshalkothavade]

My OTRS version is 6.0.6.

My server configuration is:
11 GB ram, 6 core processor, ORACLE Linux as OS -- Application Server
Database is ORACLE 12C, AIX OS -- 25 GB RAM -- Database Server

Daily inflow of ticket size is 2K to 3K.

Concurrent users count is 10 to 30.

We are getting slowness, not able to understand the exact cause of slowness.

With 1 user Can see 2-5 seconds to load page but with 10 users it takes approximately 20 to 40 seconds. Sometime it takes 1 minute.

While identifying the cause at few places (like listing and pagination) code was creating a issue. We fixed that and able to save 8 seconds in load time. (I will share update on that part later) But still 12 to 32 seconds is very high.

Does anyone also come with similar situation before? Want to see your suggestions on improving performance. Is the performance same for all of you?

Note: we have done few customization on Agent dashboard.

[root]

Hi,

Are there any external customer user databases connected with DB or LDAP backend?

- Roy

[zzz]

Hello,

That performance ist definitely not normal. Especially for your server resources.

Things you could check:
- DNS problems?
- You can temporarily active the performance log (admin area) to see if all pages are equally slow or just selected ones.
- What does the Google Chrome Network tab say? Does is take a long time to load static content, and what does the waterfall say?

If you want to take it to the next level, you could profile your code with NYTProf to see which parts are slowing down the request.

It could also be the database/database connection.

Best regards
Emin

[harshalkothavade]

Thank you everyone for quick reply:

see my comments here:

- DNS problems?
-- We tried accessing directly by hostname and IP - found same observations.


- You can temporarily active the performance log (admin area) to see if all pages are equally slow or just selected ones.
-- Performance log also shows slowness as user base increases.

- What does the Google Chrome Network tab say? Does is take a long time to load static content, and what does the waterfall say?
Google chrome tab also shows slowness. major time taken is in getting response from the server. loading static content happens very quick.

NYTProf
Will take your suggestion. This is new for me. I will run my code through this and will update you here.

Can you suggest how do I check if DB connection could be problem? or DB is problem here?

Someone in other posts mentioned that ORACLE DB is slow with OTRS. Is this true?

[harshalkothavade]

No we are not connecting to external database for customer information. only external agency here is email box which is being read by cron jobs scheduled. we have approx 10 mailboxes configured and every minute we are fetching those in parallel.

To see if any cron job creates trouble, we stopped all cron jobs and response was still same...

In DB we have around 90000 tickets with 5 articles on an average for each ticket.

[zzz]

90k tickets is close to none, that shouldn't be the issue.
I don't have any benchmarks to hand but using Oracle for OTRS, in my humble opinion, is an overkill.

OTRS doesn't make use of database functions (by design) that would justify a license.
The performance log could give hints about the database performance. Check if pages with a small number of SQL commands run significantly faster.
The console command 'Dev::Tools::CacheBenchmark' could also be helpful (or errors in the support collector).

Best regards
Emin

[root]

Hi,

Do you have escalations configured?

- Roy

[harshalkothavade]

Yes we have escalations configured. Its business mandate. we can not move it out. But we are thinking of making it DB procedure and removing overhead from application.

[harshalkothavade]

What does Dev::Tools::CacheBenchmark ?

Not sure what this is going to help me. I can not make any judgement on this. Any suggestions? I have updated my stats below.

Here is my Stats Display

cache benchmark.png

[zzz]

Checking the read/write speed. Looks inconspicuous if ran while a lot of users where working on the system.

Best regards
Emin

[root]

Hi,

Did you ever try to disable your dashboard customizations to exclude them as the origin of your performance issue?

- Roy

[harshalkothavade]

what we observed that otrs 6.0.6 has performance issue.

To verify we deployed both(6.0.6 and 6.0.24) vanilla version on two separate instances and observed the difference, we saw there is a significant difference.

We inspected our code too and found no reason to get it slower. So we migrated to latest 6.0.24 and merging our modified code. Will keep you posted here.

But haven't understood about cache benchmark -- what diagnosis information it is going to give us..? any update on that part. I have posted snapshot of my cache benchmark result in earlier post here.

[zzz]

As mentioned above, it gives information about the read and write speed of your cache, which, in some scenarios, can be the reason for a slow system.

The reasons could be a slow hard drive and a bad caching code; especially if you're writing your own modules.

Your operations per second look pretty good which indicates that it's most likely something else.

Best regards
Emin

[harshalkothavade]

Ok Thanks for the details.

After moving to 6.0.24 seems performance has given us good shape but still not as expected with configuration of infra we are using.

what's worst is when 30 users trying to login (login + agent dashboard) in one second (through JMETER) it takes 90 seconds in average.
with 10 users it goes upto 32 seconds.
with 5 users 20 seconds
with single user it takes 8 second.

But after login if users are loading pages they load in good way: e.g. within 5-10 seconds agent dashboard page get loaded even if there are 30 concurrent requests.

[zzz]

That still sounds pretty bad and you should aim at <1 second on the dashboard.

Check the tips Roy and I gave.

Best regards
Emin

[harshalkothavade]

IS perl DBD::ORACLE is real issue here?

As you guys suggested I run NYTpro and found that DBD::ORACLE was showing 168 MS for single request.

There are other section by this is one seems to be a troublesome...

[zzz]

As suspected, it looks like it's the DB. Check the DB performance logs and if DBD::ORACLE is up to date.
It could also be a problem with the connection to the DB server.

[wurzel]

Hi,

the perl pre-loading in
apache2-perl-startup.pl

Code: Select all

# enable this if you use oracle
#use DBD::Oracle ();
#use Kernel::System::DB::oracle;

is active?

Flo

[schulmann]

Hi,

maybe it's a problem of your apache configuration.

My rule of thumb is

• KeepAlive Off
• 5 apache processes (MaxClients)
• for every 5 concurrent agents 1 additional apache process
• for every apache process at least 250 MB RAM

Claus-Dieter

[harshalkothavade]

Thanks for your replies....

Have anyone used connection pooling with oracle in OTRS?

Does anyone know where we can configure this?

[root]

Hi,

Did you uncomment these lines in the file scripts/apache2-perl-startup.pl?

Code: Select all

#use DBD::Oracle ();
#use Kernel::System::DB::oracle;

- Roy