OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Moderator: crythias

Post Reply
manoj99
Znuny newbie
Posts: 41
Joined: 22 Jan 2020, 00:00
Znuny Version: 5.0.42
Real Name: mk99

OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by manoj99 »

Hello,

I'm trying to migrate Article Storage from DB to FS and I ran the below script
shell> bin/otrs.Console.pl Admin::Article::StorageSwitch --target ArticleStorageFS

It took my almost half a day to get this script done. However, I noticed that, the Article table was shrunk drastically but not the article_attachment.

Also, I see /opt/otrs/var/article filled with all the attachments.

I wanted to know if there is anything that I'm missing during this process.

Please note, I have manually changed in the StorageModule setting via SysConfig and also mentioned the below in Config.pm

$Self->{'Ticket::StorageModule'} = 'Kernel::System::Ticket::ArticleStorageFS';
$Self->{'Ticket::StorageModule::CheckAllBackends'} = '1';


Please let me know if anyone has any idea or faced any similar situation.

Thanks,
Manoj.
manoj99
Znuny newbie
Posts: 41
Joined: 22 Jan 2020, 00:00
Znuny Version: 5.0.42
Real Name: mk99

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by manoj99 »

Hi,

I retried to move Article Storage from ArticleStorageDB to ArticleStorageFS using the following script by cleaning up the previously moved articles.

shell> bin/otrs.Console.pl Admin::Article::StorageSwitch --target ArticleStorageFS.

I have mentioned below modules in config.pm and the GUI sysconfig reflects similar changes.

$Self->{'Ticket::StorageModule'} = 'Kernel::System::Ticket::ArticleStorageFS';
$Self->{'Ticket::StorageModule::CheckAllBackends'} = '1';

Although, the script ran and completed, none of the articles moved to FS this time.

/var/article only has the latest folder i.e 2020 but nothing else.

I turned on the debug mode, it seems that for some reason the default value is getting set i.e ArticleStorageDB though ArticleStorageFS is mentioned in config.pm

Code: Select all

Debug: Config.pm ->Get('Ticket::NumberGenerator') --> Kernel::System::Ticket::Number::DateChecksum
Debug: Config.pm ->Get('Ticket::IndexModule') --> Kernel::System::Ticket::IndexAccelerator::StaticDB
Debug: Config.pm ->Get('Ticket::StorageModule') --> Kernel::System::Ticket::ArticleStorageFS
Debug: Config.pm ->Get('Ticket::StorageModule::CheckAllBackends') --> 1
Debug: Config.pm ->Get('Ticket::SearchIndexModule') --> Kernel::System::Ticket::ArticleSearchIndex::StaticDB
Debug: Config.pm ->Get('Ticket::CustomModule') --> <undef>
Debug: Config.pm ->Get('ArticleDir') --> /opt/otrs/var/article
Debug: Config.pm ->Get('Cache::Module') --> Kernel::System::Cache::FileStorable
Debug: Config.pm ->Get('DynamicFields::Driver') --> HASH(0x3288de0)
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Text') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Checkbox') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Date') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::DateTime') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Dropdown') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Multiselect') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Text') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::Text') --> <undef>
Debug: Config.pm ->Get('DynamicFields::Extension::Driver::TextArea') --> <undef>
Debug: Config.pm ->Get('DynamicFields::ObjectType') --> HASH(0x3289158)
Debug: Config.pm ->Get('DynamicFields::Extension::Backend') --> <undef>
Debug: Config.pm ->Get('Ticket::ArchiveSystem') --> 1
1/360361 (TicketID:1)
Debug: Config.pm ->Get('Ticket::EventModulePost') --> HASH(0x33c94e8)
Debug: Config.pm ->Set(Key => Ticket::StorageModule, Value => Kernel::System::Ticket::ArticleStorageDB)
Debug: Config.pm ->Get('Ticket::NumberGenerator') --> Kernel::System::Ticket::Number::DateChecksum
Debug: Config.pm ->Get('Ticket::IndexModule') --> Kernel::System::Ticket::IndexAccelerator::StaticDB
Debug: Config.pm ->Get('Ticket::StorageModule') --> Kernel::System::Ticket::ArticleStorageDB
Debug: Config.pm ->Get('Ticket::StorageModule::CheckAllBackends') --> 1
Debug: Config.pm ->Get('Ticket::SearchIndexModule') --> Kernel::System::Ticket::ArticleSearchIndex::StaticDB
Debug: Config.pm ->Get('Ticket::CustomModule') --> <undef>
Debug: Config.pm ->Get('ArticleDir') --> /opt/otrs/var/article
Debug: Config.pm ->Set(Key => Ticket::StorageModule, Value => Kernel::System::Ticket::ArticleStorageDB)
Debug: Config.pm ->Get('Ticket::NumberGenerator') --> Kernel::System::Ticket::Number::DateChecksum
Debug: Config.pm ->Get('Ticket::IndexModule') --> Kernel::System::Ticket::IndexAccelerator::StaticDB
Debug: Config.pm ->Get('Ticket::StorageModule') --> Kernel::System::Ticket::ArticleStorageDB
Debug: Config.pm ->Get('Ticket::StorageModule::CheckAllBackends') --> 1
Debug: Config.pm ->Get('Ticket::SearchIndexModule') --> Kernel::System::Ticket::ArticleSearchIndex::StaticDB
Debug: Config.pm ->Get('Ticket::CustomModule') --> <undef>
Debug: Config.pm ->Get('ArticleDir') --> /opt/otrs/var/article
Debug: Config.pm ->Get('Ticket::StorageModule::CheckAllBackends') --> 1
2/360361 (TicketID:21)
I did clean up the cache and rebuild the system but that didn't help.

bin/otrs.Console.pl Maint::Config::Rebuild
bin/otrs.Console.pl Maint::Cache::Delete


Any help is highly appreciated.

Thanks,
Manoj.
zzz
Znuny superhero
Posts: 888
Joined: 15 Dec 2016, 15:13
Znuny Version: All
Real Name: Emin
Company: Efflux GmbH
Contact:

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by zzz »

Hello Manoj,

Double-check that the permissions are set correctly.
Any output on the error log of your web server?

If nothing seems to help, you should try an update to the latest version (6) and give it another try.

Best regards
Emin
Professional OTRS, Znuny & OTOBO services: efflux.de | efflux.de/en/

Free and premium add-ons: German | English
manoj99
Znuny newbie
Posts: 41
Joined: 22 Jan 2020, 00:00
Znuny Version: 5.0.42
Real Name: mk99

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by manoj99 »

Thanks Emin for the response.

I have 750 permissions on /opt/otrs/var with otrs as user and apache as the group

On article folder I have sticky bit.. so seems like the permissions are fine.

Like I mentioned earlier, the debug statement mentions ArticleStorageDB, though I have given ArticleStorageFS in config.pm.

For some reason, the ArticleStorageFS parameter value is getting overwrite. (please check my previous email )

I don't see any error logs on apache and otrs.

Upgrading to v6 is also one the major issues I have but that's an other story to discuss.

Best,
Manoj.
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by root »

Hi Manoj,

What is the output if you just start migrating old tickets:

Code: Select all

bin/otrs.Console.pl Admin::Article::StorageSwitch --tickets-closed-before-days 1000 --target ArticleStorageFS --tolerant
Please adjust the 1000 to a value what matches your system.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
zzz
Znuny superhero
Posts: 888
Joined: 15 Dec 2016, 15:13
Znuny Version: All
Real Name: Emin
Company: Efflux GmbH
Contact:

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by zzz »

Permissions should only be set with the otrs.SetPermissions.pl script unless you know you're doing.
Don't change the Config.pm. Use the system configuration instead for these kinds of settings.

StorageModule is for new articles. Even if set wrong, it wouldn't explain why the articles don't move when you execute "bin/otrs.Console.pl Admin::Article::StorageSwitch" and don't receive errors.

— Emin
Professional OTRS, Znuny & OTOBO services: efflux.de | efflux.de/en/

Free and premium add-ons: German | English
root
Administrator
Posts: 3934
Joined: 18 Dec 2007, 12:23
Znuny Version: Znuny and Znuny LTS
Real Name: Roy Kaldung
Company: Znuny
Contact:

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by root »

Manoj,

I just checked your post again and have some questions:

1.) You wrote "Also, I see /opt/otrs/var/article filled with all the attachments." and I assume the data was properly migrated.
2.) When you said "I noticed that the Article table was shrunk drastically but not the article_attachment." how did you check this? If you use InnoDB did you have innodb_file_per_table active?
3.) "I retried to move Article Storage from ArticleStorageDB to ArticleStorageFS using the following script by cleaning up the previously moved articles." Did you run this on the same system? Sometimes the database could stay the old size bu does not contain any data, just internally deleted records.

- Roy
Znuny and Znuny LTS running on CentOS / RHEL / Debian / SLES / MySQL / PostgreSQL / Oracle / OpenLDAP / Active Directory / SSO

Use a test system - always.

Do you need professional services? Check out https://www.znuny.com/

Do you want to contribute or want to know where it goes ?
manoj99
Znuny newbie
Posts: 41
Joined: 22 Jan 2020, 00:00
Znuny Version: 5.0.42
Real Name: mk99

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by manoj99 »

root wrote: 26 Oct 2020, 20:11 Hi Manoj,

What is the output if you just start migrating old tickets:

Code: Select all

bin/otrs.Console.pl Admin::Article::StorageSwitch --tickets-closed-before-days 1000 --target ArticleStorageFS --tolerant
Please adjust the 1000 to a value what matches your system.

- Roy
Hi Roy,

I have executed the script you mentioned,

I have the following output

1/328321 (TicketID:1)
2/328321 (TicketID:21)
3/328321 (TicketID:41)
4/328321 (TicketID:42)
5/328321 (TicketID:61)
6/328321 (TicketID:81)
7/328321 (TicketID:82)
8/328321 (TicketID:83)
...
...
...
328321/328321
done.

the articles have not been moved to /var/article although the script is not showing any error nor I did not find anything on error logs

Thanks,
Manoj.
Last edited by manoj99 on 26 Oct 2020, 21:45, edited 1 time in total.
manoj99
Znuny newbie
Posts: 41
Joined: 22 Jan 2020, 00:00
Znuny Version: 5.0.42
Real Name: mk99

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by manoj99 »

zzz wrote: 26 Oct 2020, 20:13 Permissions should only be set with the otrs.SetPermissions.pl script unless you know you're doing.
Don't change the Config.pm. Use the system configuration instead for these kinds of settings.

StorageModule is for new articles. Even if set wrong, it wouldn't explain why the articles don't move when you execute "bin/otrs.Console.pl Admin::Article::StorageSwitch" and don't receive errors.

— Emin
Hi Emin,

I did set the permissions using otrs.SetPermissions.pl.

I agree with you, setting StorageModule on Config.pm is for new modules , but in my case, although I did mention in sysconfig, the ArticleStorageFS parameter is getting overwritten by default one i.e ArticleStorageDB.

Manoj.
manoj99
Znuny newbie
Posts: 41
Joined: 22 Jan 2020, 00:00
Znuny Version: 5.0.42
Real Name: mk99

Re: OTRS V5 Ticket::StorageModule Migration from ArticleStorageDB to ArticleStorageFS

Post by manoj99 »

root wrote: 26 Oct 2020, 20:18 Manoj,

I just checked your post again and have some questions:

1.) You wrote "Also, I see /opt/otrs/var/article filled with all the attachments." and I assume the data was properly migrated.
2.) When you said "I noticed that the Article table was shrunk drastically but not the article_attachment." how did you checked this? If you use InnoDB did you have innodb_file_per_table active?
3.) "I retried to move Article Storage from ArticleStorageDB to ArticleStorageFS using the following script by cleaning up the previously moved articles." Did you run this on the same system? Sometimes the database could stay the old size bu does not contain any data, just interbally deleted records.

- Roy

Hi Roy,

1. When I first ran the script, the articles moved under /var/article without any errors but only migrated article table not the article_attachment.

2. We use Oracle database, I compared the table size before and after using sql describe queries

3. I ran on the same system by doing Oracle db resync and cleaned up the previous articles that were migrated to check if that can migrate both article and article_attachment table articles. But, as mentioned earlier, the script shows the progress status but none of them move (as I see in debug logs, the parameter ArticleStorageFS mentioned in sys config is getting overwritten by ArticleStorageDB from somewhere.

Is there any possibility after the db resync, the default parameter is stored somewhere in the database?

Thanks,
Manoj.
Post Reply