Email Character Conversion Problem

Moderator: crythias

Post Reply
yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Email Character Conversion Problem

Post by yourhandisonmyleg » 09 Aug 2011, 09:13

I am having a problem displaying Chinese in some messages in the OTRS system. Most of the email's Chinese characters get converted to garbage or different characters. It isn't consistently happening, and it is specific to certain customers.
This... 請以手動調正Exposure Time到1/120或其他參數,確認是否還有閃爍
Becomes this in OTRS... ��以手�诱{正Exposure Time到1/120或其他���担��_�J是否��有�W�q

My colleague's reply (using Outlook) was also trashed (sent via Outlook, not OTRS... still testing the system)... so, it appears specific to that email's character set/font. It appears to be gb2313 and simsun, but another email received today (without a problem) also appears to use the same char set and font (saved the email as html and used browser to get char set). The emails are all ok in Outlook.

Encode::HanExtra isn't installed. I have tried from rpm and manual install, but encountering a problem which I posted on another site as I get some errors on the make test which I am not sure what to do with. Here it is if it is relevant...
http://www.linuxquestions.org/questions ... ra-896283/

Perl 5.12.4 is installed. Naturally I would like to get HanExtra installed to rule that out, but I am not sure how long sorting that out will take (don't have hoards of people using it seems)... and I am not sure that is the issue since other emails with the same char set (I think) are coming through ok.

We are in Taiwan (traditional characters) and this customer is in the mainland I think. However, his colleague is also sending email (using simplified characters) and OTRS displays that ok.

May ask my colleague to post in the Chinese section when we have a chance.
Thx
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg » 10 Aug 2011, 03:20

I got HanExtra installed...
o Encode::HanExtra...............ok (v0.23)

But I tried resending a problem email, and the problem persists.

Any ideas? Thx
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg » 10 Aug 2011, 07:11

Actually, it appears that OTRS (or some related component) isn't handling traditional Chinese characters. After more testing, emails with traditional characters consistently aren't rendered properly.

Simplified Chinese seems to come through ok.

Core DefaultUsedLanguages has zh_CN and zh_TW
/usr/lib/perl5/Encode has TW.pm as well as CN.pm (and a CN directory with HZ.pm)
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg » 15 Aug 2011, 05:06

after some more use, the traditional characters sometimes come through ok.
it seems to depend on the email. i haven't yet seen a case where simplified gets trashed.
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

micshi
OTRS wizard
Posts: 191
Joined: 05 Jul 2011, 16:10
OTRS Version?: 3.2.2
Real Name: Micshi
Location: Beijing
Contact:

Re: Email Character Conversion Problem

Post by micshi » 15 Aug 2011, 14:38

Hi,

By default, Perl will not support Simplified Chinese and Traditional Chinese "Encode" even if Encode::HanExtra was installed. Since the following line items are still commented out by default in Perl's configuration file of "Config.pm", which file maybe located in the directory of "/usr/lib/perl/5.x.y/Encode/" or "usr/perl/perl/5.x.y/Encode/" depending on your linux type.

#'big5plus' => 'Encode::HanExtra',
#'euc-tw' => 'Encode::HanExtra',
#'gb18030' => 'Encode::HanExtra',

You might remove those comment character "#" in the Config.pm for correct Perl configuration.

micshi
OTRS 3.2.2, ITSM 3.2.2, SystemMonitor
Ubuntu, Apache2, MySQL 5
My OTRS Blog
OTRS Basic Concept Training
OTRS Demo Instance

yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg » 17 Aug 2011, 06:47

thx for the reply.
i saw the lines commented out in Config.pm. uncommented them and restarted apache. however, i noticed that some of the supporting pm files (e.g. HanExtra.pm) weren't in the Encode directory which led me to think there was a problem with the HanExtra install.

it appears that fedora normally doesn't use /usr/local and that is where the HanExtra files went. anyways, i am sorting that out now... after some changes, ran the make install again and noted that all files went to /usr/lib which apparently is "normal" for fedora. i think that should about do it, but still checking on 1 thing.

anyways will see if the latest changes have any effect.
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg » 17 Aug 2011, 08:59

looks like HanExtra is installed ok now. perl -MEncode::HanExtra -e 1 yields no errors... and pm files are where i would expect them.

however, i still see the problem with traditional chinese characters getting converted in some emails... not all.
mysql characterset is utf8 unicode. i see the data in a_subject & a_body has the corrupted characters. created a test table and successfully inserted data (copy/paste from problem email data). probably not a bulletproof test, but doesn't look like a db issue.

not sure what else i can do here. perhaps file a bug?
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

yourhandisonmyleg
OTRS newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
OTRS Version?: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg » 19 Aug 2011, 05:05

searched the bug database, and i found a bug (bug 4201) which looks like the same problem. unfortunately, it was opened 2yrs ago, and never progressed beyond identifying it as a potential defect.
anyways, i updated the bug requesting an update on the status.
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL

briancolton1
OTRS newbie
Posts: 1
Joined: 25 Nov 2011, 16:23
OTRS Version?: 2.1
Real Name: brian colton

Re: Email Character Conversion Problem

Post by briancolton1 » 26 Nov 2011, 10:53

yup that's unfortunately bad news, is it still open ?

nuera
OTRS newbie
Posts: 16
Joined: 27 May 2012, 12:21
OTRS Version?: 3.1.5

Re: Email Character Conversion Problem

Post by nuera » 06 Jun 2012, 05:58

Is there any schedule for this bug? It is very important for some international companies in China.

Post Reply