Email Character Conversion Problem

Moderator: crythias

Post Reply
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Email Character Conversion Problem

Post by yourhandisonmyleg »

I am having a problem displaying Chinese in some messages in the OTRS system. Most of the email's Chinese characters get converted to garbage or different characters. It isn't consistently happening, and it is specific to certain customers.
This... 請以手動調正Exposure Time到1/120或其他參數,確認是否還有閃爍
Becomes this in OTRS... ��以手�诱{正Exposure Time到1/120或其他���担��_�J是否��有�W�q

My colleague's reply (using Outlook) was also trashed (sent via Outlook, not OTRS... still testing the system)... so, it appears specific to that email's character set/font. It appears to be gb2313 and simsun, but another email received today (without a problem) also appears to use the same char set and font (saved the email as html and used browser to get char set). The emails are all ok in Outlook.

Encode::HanExtra isn't installed. I have tried from rpm and manual install, but encountering a problem which I posted on another site as I get some errors on the make test which I am not sure what to do with. Here it is if it is relevant...
http://www.linuxquestions.org/questions ... ra-896283/

Perl 5.12.4 is installed. Naturally I would like to get HanExtra installed to rule that out, but I am not sure how long sorting that out will take (don't have hoards of people using it seems)... and I am not sure that is the issue since other emails with the same char set (I think) are coming through ok.

We are in Taiwan (traditional characters) and this customer is in the mainland I think. However, his colleague is also sending email (using simplified characters) and OTRS displays that ok.

May ask my colleague to post in the Chinese section when we have a chance.
Thx
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg »

I got HanExtra installed...
o Encode::HanExtra...............ok (v0.23)

But I tried resending a problem email, and the problem persists.

Any ideas? Thx
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg »

Actually, it appears that OTRS (or some related component) isn't handling traditional Chinese characters. After more testing, emails with traditional characters consistently aren't rendered properly.

Simplified Chinese seems to come through ok.

Core DefaultUsedLanguages has zh_CN and zh_TW
/usr/lib/perl5/Encode has TW.pm as well as CN.pm (and a CN directory with HZ.pm)
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg »

after some more use, the traditional characters sometimes come through ok.
it seems to depend on the email. i haven't yet seen a case where simplified gets trashed.
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
micshi
Znuny expert
Posts: 191
Joined: 05 Jul 2011, 16:10
Znuny Version: 3.2.2
Real Name: Micshi
Location: Beijing
Contact:

Re: Email Character Conversion Problem

Post by micshi »

Hi,

By default, Perl will not support Simplified Chinese and Traditional Chinese "Encode" even if Encode::HanExtra was installed. Since the following line items are still commented out by default in Perl's configuration file of "Config.pm", which file maybe located in the directory of "/usr/lib/perl/5.x.y/Encode/" or "usr/perl/perl/5.x.y/Encode/" depending on your linux type.

#'big5plus' => 'Encode::HanExtra',
#'euc-tw' => 'Encode::HanExtra',
#'gb18030' => 'Encode::HanExtra',

You might remove those comment character "#" in the Config.pm for correct Perl configuration.

micshi
OTRS 3.2.2, ITSM 3.2.2, SystemMonitor
Ubuntu, Apache2, MySQL 5
My OTRS Blog
OTRS Basic Concept Training
OTRS Demo Instance
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg »

thx for the reply.
i saw the lines commented out in Config.pm. uncommented them and restarted apache. however, i noticed that some of the supporting pm files (e.g. HanExtra.pm) weren't in the Encode directory which led me to think there was a problem with the HanExtra install.

it appears that fedora normally doesn't use /usr/local and that is where the HanExtra files went. anyways, i am sorting that out now... after some changes, ran the make install again and noted that all files went to /usr/lib which apparently is "normal" for fedora. i think that should about do it, but still checking on 1 thing.

anyways will see if the latest changes have any effect.
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg »

looks like HanExtra is installed ok now. perl -MEncode::HanExtra -e 1 yields no errors... and pm files are where i would expect them.

however, i still see the problem with traditional chinese characters getting converted in some emails... not all.
mysql characterset is utf8 unicode. i see the data in a_subject & a_body has the corrupted characters. created a test table and successfully inserted data (copy/paste from problem email data). probably not a bulletproof test, but doesn't look like a db issue.

not sure what else i can do here. perhaps file a bug?
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
yourhandisonmyleg
Znuny newbie
Posts: 10
Joined: 12 Jul 2011, 06:09
Znuny Version: 3.0.5

Re: Email Character Conversion Problem

Post by yourhandisonmyleg »

searched the bug database, and i found a bug (bug 4201) which looks like the same problem. unfortunately, it was opened 2yrs ago, and never progressed beyond identifying it as a potential defect.
anyways, i updated the bug requesting an update on the status.
OTRS 3.0.9
Fedora 15 (2.6.38.6-26.rc1.fc15.i686)
Apache, MySQL
briancolton1
Znuny newbie
Posts: 1
Joined: 25 Nov 2011, 16:23
Znuny Version: 2.1
Real Name: brian colton

Re: Email Character Conversion Problem

Post by briancolton1 »

yup that's unfortunately bad news, is it still open ?
nuera
Znuny newbie
Posts: 16
Joined: 27 May 2012, 12:21
Znuny Version: 3.1.5

Re: Email Character Conversion Problem

Post by nuera »

Is there any schedule for this bug? It is very important for some international companies in China.
Post Reply