Investigation into getting Chinese EPG in OpenPLi/enigma2

**i1646tvk84423** · 17th January, 2013, 09:39 AM

I've succeeded in getting Chinese EPG (specifically from SG Starhub) on OpenPLi 2.1 & enigma2, and I thought I share what I found and did. I'm a newbie on the scene so don't take what I say as the absolute truth. I'm still investigating and learning.

Side note: I understand cnigma2 (created by http://freedmx.net/?79) is a mod of enigma2 to support Chinese, but all I can find are .nfi binary images. Called me paranoid but I simply don't trust software from the scene without source code. (This is the reason I chose OpenPLi in the first place.)

-- Part 1 --

Now, embedded in the DVB streams is something called DVB-SI (service information), which I read up @ http://www.dvb.org/technology/standa...468v1.12.1.pdf.

Side note: From section 5.2.5, you can see how your STB get its time. Using dvbsnoop (http://dvbsnoop.sourceforge.net/), I observe a TDT (time and date table) packet every 3 seconds.

Using dvbsnoop (see http://dvbsnoop.sourceforge.net/exam...ample-eit.html for EIT (event info table) examples), for a Chinese programme, I see SH broadcast two short event descriptors (see section 6.2.37), one for English and one for Chinese. The ISO639 lang code for Chinese is "chi".

For non-English (or non-Latin lang), the "event_name_char" and "text_char" are to be encoded as specified in Annex A.

In Annex A.2, it says that if the first byte is less than 0x20, it denotes the encoding to be used, otherwise it is normal data.

SH violates this aspect. When the data is Chinese, it is sending it as UTF8 without putting a extra "0x15" byte (see table A.3) in front. Our STB, adhering to the spec, (including dvbsnoop) simply assumes it is normal data and uses Latin charset (ISO8859) to display it.

Example: The character "ju" is, as shown in http://www.scarfboy.com/coding/unicode-tool?s=E589A7 is encoded by three octets: 0xe5 0x89 0xa7. If the STB is aware they're UTF8, it will display these three octets as a single character; if it doesn't it will display these three octets as three separate characters, which is the case currently.

-- End of Part 1 --

**i1646tvk84423** · 17th January, 2013, 10:12 AM

-- Part 2 --

As you know, enigma2 is the key app of our STB and controls many aspects including web server, interfacing with DVB drivers, etc. It is not the OS. The OS is GNU/Linux (which means it has a Linux kernel and GNU supporting utilities). enigma2 sits on top of this and is the main app.

It has a C++ core and a Python plugin framework.

As of around Oct 2011, DMM decides to close-source enigma2 (and also transition to enigma3 (closed source)?). DMM's enigma2 source is @ http://git.opendreambox.org/?p=enigma2.git;a=summary which you'll notice is absent of any .cpp source codes, only the include header files are available for Python plugin authors.

OpenPLi took the last available enigma2 source and fork it @ http://openpli.git.sourceforge.net/g...gma2;a=summary

Coming back to our objective, EITs are streaming continuously, so for this reason, enigma2 needs to use its C++ core to capture these into its EPG cache.

When an EIT is received, epgcache.cpp line 106 would lookup the ISO639 3-char encoding (function getCountryCodeDefaultMapping), while lines 112 & 113 calls a function called convertDVBUTF8.

getCountryCodeDefaultMapping is defined in encoding.cpp line 86. You can see the mapping is taken from a config files in /etc/enigma2/encoding.conf in your STB. Lines 39, 50, 53 and 61 shows the four available formats acceptable in encoding.conf.

convertDVBUTF8 is defined in estring.cpp at line 370. If SH follows the spec, line 423 would handle the extra 0x15 at the first, and send out the rest of the line as UTF8. Too bad, SH doesn't follow the DVB-SI spec.

Take a moment to analyze these 3 .cpp files, and you'll see that there is no way to support SH's Chinese EITs without modifying C++ code...

-- End of Part 2 --

**i1646tvk84423** · 17th January, 2013, 10:58 AM

-- Part 3 --

Since you need to modify code, now's a good time to start preparing your dev environment. Follow the steps @ http://users.telenet.be/spstefansp/openpliubuntu.html for a OpenPLi 2.1 dev environment. Don't try the 3.0, it has its own set of issues which I'm not ready to tackle right now.

Similar to http://thaidreambox.bayore.net/index.php?topic=15889, you need to add

Code:

("chi zho", _("Chinese")),

between line 326 & 327 in UsageConfig.py. Now scp the new .py files to your STB:

Code:

scp build-dm800se/tmp/sysroots/mipsel-oe-linux/usr/lib/enigma2/python/Components/UsageConfig.py root@DM800SE_IP:/usr/lib/enigma2/python/Components/

And delete the previous .pyo compiled version in your STB:

Code:

rm /usr/lib/enigma2/python/Components/UsageConfig.pyo

Restart enigma2 GUI to take effect.

Code:

kill `pidof enigma2`

Update your primary & secondary EPG language as follows:

Because SH sends English EIT for English programmes, but both Chinese and English EIT for Chinese programmes, you'll therefore want Chinese as the primary EPG language and English as the secondary language.

-- End of Part 3 --

**i1646tvk84423** · 17th January, 2013, 11:22 AM

-- Part 4 --

Add the following two lines after line 373 in estring.cpp:

Code:

if (table == 99)
return std::string((char*)data, len);

Recompile and a few enigma2 binaries will appear. Use "find" to find all of them. They should be of 2 sizes. Big 31MB ones that are not stripped and small 2.3MB ones that are stripped.

scp one of the 2.3MB stripped ones up to your STB:

Code:

scp build-dm800se/tmp/work/dm800se-oe-linux/enigma2-2.7+git0+551e8f8bc26d672d0213065789239194258bd758-r36/package/usr/bin/enigma2 root@DM800SE_IP:/usr/bin/

If scp complains file is busy, it is because your enigma2 is running. /sbin/init 4 to kill it before you scp.
Start or restart enigma2 to take effect.

Make your own encoding.conf:

Code:

cp /usr/share/enigma2/encoding.conf /etc/enigma2/

and add one line:

Code:

chi ISO8859-99

after line 9 in /etc/enigma2/encoding.conf

-- End of Part 4 --

**i1646tvk84423** · 17th January, 2013, 11:48 AM

-- Part 5 --

Final piece of the puzzle.

The .ttf font /usr/share/fonts/nmsbd.ttf does not have Chinese, so you need to replace its use with a .ttf font that has Chinese. A candidate is simsun.ttf from MS.

Becareful. Your STB flash do not have much space left after OpenPLi, something like 16.9MB left. simsun.ttf is 14.6MB.

Code:

scp simsun.ttf root@DM800SE_IP:/usr/share/fonts/

Then edit the skin definition:

Code:

cd /usr/share/enigma2/PLi-HD/
vi skin.xml

On line 15, change

Code:

<font name="Regular" filename="nmsbd.ttf" scale="100"></font>

to

Code:

<font name="Regular" filename="simsun.ttf" scale="100"></font>

Start or restart enigma2.
Voil?!

Side note: While the Chinese characters of simsun.ttf looks not bad, its English letters sux. Look at http://en.wikipedia.org/wiki/Unicode_fonts for alternatives.

-- End --

**i1646tvk84423** · 18th January, 2013, 12:02 PM

Names of channels, specified in lamedb, can also be in Chinese UTF8. They will show when you're using a Unicode font.

Latest update: Microsoft's "Arial Unicode" looks the nicest, but it is more than 23MB big, way bigger than the ~ 17MB free space left after gjstroom's .nfi image. The next best looking font is "LiHei Pro", taken from my Mac OSX, which is what I've settled on. It is only 10MB, holds less glyphs than "Arial Unicode" but good enough for SH's Chinese so far..

**i1646tvk84423** · 24th January, 2013, 10:08 AM

Saw your attached pic. Well done!!
Glad my investigation help.

**palmy** · 25th January, 2013, 05:39 AM

"My enigma2 is at https://www.dropbox.com/s/q2aiiuvu84xheus/enigma2

My UsageConfig.py is at http://www.dropbox.com/s/qabfprzj9h3dsnu/UsageConfig.py"

Thanks for the explanation. As a non-coder. Is there a way for me to make those changes on my STB? (DM800HD-se)

Hopefully I can let this Chinese EPG issue be done and over with.

Thanks

**lonari** · 26th January, 2013, 08:55 AM

Dude, great job man.
Could you get non-pli images, such as Newnigma2 to work as well?

**i1646tvk84423** · 26th January, 2013, 02:50 PM

A major reason I showed my steps is so that others can use this knowledge and apply it to any variants of enigma2 with source code.

As u can see above, the mod is just a 2-liner here and there. I'm 99.9% sure it will work on any variant of enigma2.

**lonari** · 26th January, 2013, 08:56 PM

i1646tvk84423, previously with dm800 i was able to get FULL english channel list and epg. But now with dm800se, i'm getting a couple chinese channels with garbage programme listing.

What should I do if I want full english?
using newnigma 3.3.2

**i1646tvk84423** · 27th January, 2013, 06:06 AM

Look at my msg #3 above. See the pix. Set EPG Lang 1 and 2 as English.

**Deep Impact** · 1st February, 2013, 10:47 AM

Originally Posted by i1646tvk84423

I've succeeded in getting Chinese EPG (specifically from SG Starhub) on OpenPLi 2.1 & enigma2, and I thought I share what I found and did. I'm a newbie on the scene so don't take what I say as the absolute truth. I'm still investigating and learning.

Side note: I understand cnigma2 (created by http://freedmx.net/?79) is a mod of enigma2 to support Chinese, but all I can find are .nfi binary images. Called me paranoid but I simply don't trust software from the scene without source code. (This is the reason I chose OpenPLi in the first place.)

-- Part 1 --

Now, embedded in the DVB streams is something called DVB-SI (service information), which I read up @ http://www.dvb.org/technology/standa...468v1.12.1.pdf.

Side note: From section 5.2.5, you can see how your STB get its time. Using dvbsnoop (dvbsnoop - DVB Stream Analyzer, MPEG Analyzer), I observe a TDT (time and date table) packet every 3 seconds.

Using dvbsnoop (see dvbsnoop - A DVB Stream Analyzer Tool, Example: Event Information Table for EIT (event info table) examples), for a Chinese programme, I see SH broadcast two short event descriptors (see section 6.2.37), one for English and one for Chinese. The ISO639 lang code for Chinese is "chi".

For non-English (or non-Latin lang), the "event_name_char" and "text_char" are to be encoded as specified in Annex A.

In Annex A.2, it says that if the first byte is less than 0x20, it denotes the encoding to be used, otherwise it is normal data.

SH violates this aspect. When the data is Chinese, it is sending it as UTF8 without putting a extra "0x15" byte (see table A.3) in front. Our STB, adhering to the spec, (including dvbsnoop) simply assumes it is normal data and uses Latin charset (ISO8859) to display it.

Example: The character "ju" is, as shown in Unicode codepoint lookup/search tool is encoded by three octets: 0xe5 0x89 0xa7. If the STB is aware they're UTF8, it will display these three octets as a single character; if it doesn't it will display these three octets as three separate characters, which is the case currently.

-- End of Part 1 --

Hi TS, great to see Chinese EPG in OpenPLi image

I'm using OpenPLI gjstroom for my DM800SE as well, i read ur post but i'm not really good in edit stuff and frankly speak, i don't understand at all how u get this done

can u guide me for a-z or can u upload all the necessary file that i can just use DCC to copy & overwrite existing file to get it work ? for ur info my current PLI image with all plugin @ 43~44mb.... and can u let me now all the directory for file replacement if possible... TQ

**i1646tvk84423** · 1st February, 2013, 01:09 PM

Get gjstroom's OpenPLi 2.1 DM800se image dated 2013-Jan-15 here: https://www.dropbox.com/s/39vuaxhszg...im2-ssl84b.nfi

If your gjstroom image is not dated 2013-Jan-15, there is a chance it might not work. The further the age of your image is from 2013-Jan-15, the higher chance it will not work, cos things might have changed.
Click the POWER button on your remote, to put DM800se into standby.

This is needed, cos we don't want enigma2 to be running while we're doing all these.
Download https://www.dropbox.com/s/vkh7glxoiviy8z8/encoding.conf

FTP this file "encoding" to DM800se in directory /etc/enigma2/
Download https://www.dropbox.com/s/q2aiiuvu84xheus/enigma2

FTP this file "enigma2" to DM800se in directory /usr/bin/
Download https://www.dropbox.com/s/qabfprzj9h...UsageConfig.py

FTP this file "UsageConfig.py" to DM800se in directory /usr/lib/enigma2/python/Components/
If you like bigger fonts for your EPG, download https://www.dropbox.com/s/30rg0xg6gkiia6o/skin.xml. Otherwise, to stay with the normal sized fonts, download https://www.dropbox.com/s/p453i96xgwzqayo/skin.xml.

FTP this file "skin.xml" to DM800se in directory /usr/share/enigma2/PLi-HD/
Download http://code.google.com/p/kingfont/do...iHei%20Pro.ttf
1. Rename the file from "LiHei Pro.ttf" to "liheipro.ttf"
2. FTP this file "liheipro.ttf" to DM800se in directory /usr/share/fonts/
Change your EPG language 1 to Chinese, and EPG language 2 to English, as shown:
Reboot DM800se

For extra credits, you can edit bouquet to use Chinese for Chinese channels. As far as I can tell, DreamboxEdit does not support UTF-8, so you've to use another editor.

Note: I've enlarged the fonts to cater to old folks like myself.

**Deep Impact** · 1st February, 2013, 03:04 PM

Originally Posted by i1646tvk84423

Get gjstroom's OpenPLi 2.1 DM800se image dated 2013-Jan-15 here: https://www.dropbox.com/s/39vuaxhszg...im2-ssl84b.nfi
If your gjstroom image is not dated 2013-Jan-15, there is a chance it might not work. The further the age of your image is from 2013-Jan-15, the higher chance it will not work, cos things might have changed.
Click the POWER button on your remote, to put DM800se into standby.
This is needed, cos we don't want enigma2 to be running while we're doing all these.
Download https://www.dropbox.com/s/vkh7glxoiviy8z8/encoding.conf
FTP this file "encoding" to DM800se in directory /etc/enigma2/
Download https://www.dropbox.com/s/q2aiiuvu84xheus/enigma2
FTP this file "enigma2" to DM800se in directory /usr/bin/
Download https://www.dropbox.com/s/qabfprzj9h...UsageConfig.py
FTP this file "UsageConfig.py" to DM800se in directory /usr/lib/enigma2/python/Components/
Download https://www.dropbox.com/s/30rg0xg6gkiia6o/skin.xml
FTP this file "skin.xml" to DM800se in directory /usr/share/enigma2/PLi-HD/
Download LiHei Pro.ttf - kingfont - LiHei Pro - king????? - Google Project Hosting
1. Rename the file from "LiHei Pro.ttf" to "liheipro.ttf"
2. FTP this file "liheipro.ttf" to DM800se in directory /usr/share/fonts/
Change your EPG language 1 to Chinese, and EPG language 2 to English, as shown:
Reboot DM800se

For extra credits, you can edit bouquet to use Chinese for Chinese channels. As far as I can tell, DreamboxEdit does not support UTF-8, so you've to use another editor.

Note: I've enlarged the fonts to cater to old folks like myself.

thanks for your prompt reply.... appreciate

Just wanna check with you few more things :

(1) if i don't need the big fonts size, i don't need to ftp the part (6) right ?

(2) may i know where is openpli download chinese EPG from...? i mean be default source it can get chinese EPG after mod or i still need to download some Cnigma plugin like epg helper (EPG***21161;***25163;?)

(3) DreamboxEdit does not support UTF-8... do you know any others software can support UTF-8 on english version window ?

Thanks again