Elasticsearch - errors while building index "tickets"

[lis9]

Hello!

I'm testing ES indexing for Znuny. My database is really big, (f.i: ticket_history has over 118 mln records, tickets over 11 mln), therefore testing such solution is essential.
During first indexing of ticket index I receive (in system log) three types of errors:

Fri Feb 16 09:08:48 2024 (Europe/Warsaw) error OTRS-CGI-99 Query failed for engine: Kernel::System::Search::Engine::ES. Message: [Request] ** [http://10.21.32.167:9200]-[400] Bad Request, called from sub Search::Elasticsearch::Transport::perform_request at /opt/otrs/Kernel/System/Search/Engine/ES.pm line 388. With vars: {'request' => {'path' => 'ticket/_update_by_query','serialize' => 'std','mime_type' => 'application/json','method' => 'POST','qs' => {'pipeline' => 'attachment_nested'},'body' => {'query' => {'terms' => {'TicketID' => [7762408,7762409,7762410,7762411,7762412,7762413,7762414,7762415,7762416,7762417]}}},'ignore' => []},'status_code' => 400,'body' => {'batches' => 1,'requests_per_second' => '-1','noops' => 0,'version_conflicts' => 0,'updated' => 0,'failures' => [{'cause' => {'reason' => 'runtime error','type' => 'script_exception','position' => {'offset' => 194,'start' => 167,'end' => 248},'caused_by' => {'reason' => 'Cannot cast java.lang.Integer to java.lang.String','type' => 'class_cast_exception'},'lang' => 'painless','script_stack' => ['java.base/java.lang.Class.cast(Class.java:4067)','ArticleID = Articles.get(i).ArticleID;

Fri Feb 16 09:08:47 2024 (Europe/Warsaw) error OTRS-CGI-99 Query failed for engine: Kernel::System::Search::Engine::ES. Message: [Request] ** [http://10.21.32.167:9200]-[400] Bad Request, called from sub Search::Elasticsearch::Transport::perform_request at /opt/otrs/Kernel/System/Search/Engine/ES.pm line 388. With vars: {'body' => {'timed_out' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),'deleted' => 0,'throttled_millis' => 0,'throttled_until_millis' => 0,'total' => 10,'retries' => {'bulk' => 0,'search' => 0},'failures' => [{'index' => 'ticket','cause' => {'lang' => 'painless','caused_by' => {'reason' => 'Cannot cast java.lang.Integer to java.lang.String','type' => 'class_cast_exception'},'position' => {'end' => 248,'start' => 167,'offset' => 194},'script_stack' => ['java.base/java.lang.Class.cast(Class.java:4067)','ArticleID = Articles.get(i).ArticleID;

Fri Feb 16 09:08:46 2024 (Europe/Warsaw) error OTRS-CGI-99 Query failed for engine: Kernel::System::Search::Engine::ES. Message: [Request] ** [http://10.21.32.167:9200]-[400] Bad Request, called from sub Search::Elasticsearch::Transport::perform_request at /opt/otrs/Kernel/System/Search/Engine/ES.pm line 388. With vars: {'request' => {'ignore' => [],'qs' => {'pipeline' => 'attachment_nested'},'body' => {'query' => {'terms' => {'TicketID' => [7762438,7762439,7762440,7762441,7762442,7762443,7762444,7762445,7762446,7762447]}}},'mime_type' => 'application/json','method' => 'POST','serialize' => 'std','path' => 'ticket/_update_by_query'},'status_code' => 400,'body' => {'deleted' => 0,'timed_out' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),'retries' => {'search' => 0,'bulk' => 0},'throttled_until_millis' => 0,'total' => 10,'throttled_millis' => 0,'took' => 51,'updated' => 0,'failures' => [{'status' => 400,'cause' => {'reason' => 'runtime error','type' => 'script_exception','script' => ' ...','caused_by' => {'reason' => 'Cannot cast java.lang.Integer to java.lang.String','type' => 'class_cast_exception'},'lang' => 'painless','position' => {'end' => 248,'start' => 167,'offset' => 194},'script_stack' => ['java.base/java.lang.Class.cast(Class.java:4067)','ArticleID = Articles.get(i).ArticleID;

Is this ES plugin problem or maybe my ES engine is misconfigured?

Daniel Lisiecki

[Johannes]

I would say, if you have millions of tickets, so many more millions of article, then 3 errors is is totally fine.
There may be old tickets where some references are missing.

Maybe someone deleted articles in the database and forgot references.

[lis9]

Thanks for your reply. Unfortunately, there aren't just these 3 errors. Those are 3 TYPES of errors

Currenly I am trying to run indexing on production environment, and constantly receiving such error (for article index):
Message: Query failed for engine: Kernel::System::Search::Engine::ES. Message: [Request] ** [https://xxxxxx:9200]-[400] Bad Request, called from sub Search::Elasticsearch::Transport::perform_request at /opt/otrs/Kernel/System/Search/Engine/ES.pm line 388. With vars: {'status_code' => 400,'request' => {'qs' => {'pipeline' => 'attachment_nested'},'path' => 'ticket/_update_by_query','body' => {'query' => {'terms' => {'TicketID' => [9952883]}}},'method' => 'POST','ignore' => [],'mime_type' => 'application/json','serialize' => 'std'},'body' => {'throttled_millis' => 0,'timed_out' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),'deleted' => 0,'retries' => {'bulk' => 0,'search' => 0},'throttled_until_millis' => 0,'version_conflicts' => 0,'noops' => 0,'batches' => 1,'failures' => [{'status' => 400,'index' => 'ticket','cause' => {'reason' => 'runtime error','type' => 'script_exception','caused_by' => {'type' => 'class_cast_exception','reason' => 'Cannot cast java.lang.Integer to java.lang.String'},'position' => {'start' => 167,'end' => 248,'offset' => 194},'script_stack' => ['java.base/java.lang.Class.cast(Class.java:4067)','ArticleID = Articles.get(i).ArticleID;
ArrayList ',' ^---- HERE'],'script' => ' ...','lang' => 'painless'},'id' => '9952883'}],'total' => 1,'took' => 7,'requests_per_second' => '-1','updated' => 0}}


Traceback (1575345):
Module: Kernel::System::Search::Engine::ES::QueryExecute Line: 204
Module: Kernel::System::Search::Object::Engine::ES::Ticket::ObjectIndexGeneric Line: 1507
Module: Kernel::System::Search::Object::Engine::ES::Ticket::ObjectIndexUpdate Line: 1024
Module: Kernel::System::Search::ObjectIndexUpdate Line: 564
Module: Kernel::System::Search::Object::Base::ObjectIndexQueueHandle Line: 1203
Module: Kernel::System::Console::Command::Maint::Search::ES::IndexQueueDataProcess::Run Line: 97
Module: (eval) Line: 461
Module: Kernel::System::Console::BaseCommand::Execute Line: 455
Module: (eval) Line: 143
Module: Kernel::System::Daemon::DaemonModules::SchedulerTaskWorker::Cron::Run Line: 122
Module: Kernel::System::Daemon::DaemonModules::SchedulerTaskWorker::Run Line: 236
Module: (eval) Line: 332
Module: main::Start Line: 332
Module: /opt/otrs/bin/otrs.Daemon.pl Line: 153


This is: "Cannot cast java.lang.Integer to java.lang.String'".

My opinion is the problem exists when somebody updates ticket and adds an article. Indexing new article -> updating index causes error. the new articles can't see in kibana. Error happens for every new ticket created in system.

Any idea why does this happen?

[Johannes]

Please open an Issue on GitHub if you think this is is a bug and not related to your setup.

https://github.com/znuny/Znuny-Search/issues

Thanks