Investigation into getting Chinese EPG in OpenPLi/enigma2
Register
Page 1 of 3 123 LastLast
Results 1 to 15 of 40
  1. #1
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default Investigation into getting Chinese EPG in OpenPLi/enigma2

    I've succeeded in getting Chinese EPG (specifically from SG Starhub) on OpenPLi 2.1 & enigma2, and I thought I share what I found and did. I'm a newbie on the scene so don't take what I say as the absolute truth. I'm still investigating and learning.

    Side note: I understand cnigma2 (created by http://freedmx.net/?79) is a mod of enigma2 to support Chinese, but all I can find are .nfi binary images. Called me paranoid but I simply don't trust software from the scene without source code. (This is the reason I chose OpenPLi in the first place.)
    -- Part 1 --

    Now, embedded in the DVB streams is something called DVB-SI (service information), which I read up @ http://www.dvb.org/technology/standa...468v1.12.1.pdf.

    Side note: From section 5.2.5, you can see how your STB get its time. Using dvbsnoop (http://dvbsnoop.sourceforge.net/), I observe a TDT (time and date table) packet every 3 seconds.
    Using dvbsnoop (see http://dvbsnoop.sourceforge.net/exam...ample-eit.html for EIT (event info table) examples), for a Chinese programme, I see SH broadcast two short event descriptors (see section 6.2.37), one for English and one for Chinese. The ISO639 lang code for Chinese is "chi".

    For non-English (or non-Latin lang), the "event_name_char" and "text_char" are to be encoded as specified in Annex A.

    In Annex A.2, it says that if the first byte is less than 0x20, it denotes the encoding to be used, otherwise it is normal data.

    SH violates this aspect. When the data is Chinese, it is sending it as UTF8 without putting a extra "0x15" byte (see table A.3) in front. Our STB, adhering to the spec, (including dvbsnoop) simply assumes it is normal data and uses Latin charset (ISO8859) to display it.

    Example: The character "ju" is, as shown in http://www.scarfboy.com/coding/unicode-tool?s=E589A7 is encoded by three octets: 0xe5 0x89 0xa7. If the STB is aware they're UTF8, it will display these three octets as a single character; if it doesn't it will display these three octets as three separate characters, which is the case currently.

    -- End of Part 1 --
    Last edited by i1646tvk84423; 17th January, 2013 at 12:09 PM.

  2. The Following 2 Users Say Thank You to i1646tvk84423 For This Useful Post:

    howpeter (31st March, 2014), pirsquared8 (27th April, 2013)

  3. #2
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    -- Part 2 --

    As you know, enigma2 is the key app of our STB and controls many aspects including web server, interfacing with DVB drivers, etc. It is not the OS. The OS is GNU/Linux (which means it has a Linux kernel and GNU supporting utilities). enigma2 sits on top of this and is the main app.

    It has a C++ core and a Python plugin framework.

    As of around Oct 2011, DMM decides to close-source enigma2 (and also transition to enigma3 (closed source)?). DMM's enigma2 source is @ http://git.opendreambox.org/?p=enigma2.git;a=summary which you'll notice is absent of any .cpp source codes, only the include header files are available for Python plugin authors.

    OpenPLi took the last available enigma2 source and fork it @ http://openpli.git.sourceforge.net/g...gma2;a=summary

    Coming back to our objective, EITs are streaming continuously, so for this reason, enigma2 needs to use its C++ core to capture these into its EPG cache.

    When an EIT is received, epgcache.cpp line 106 would lookup the ISO639 3-char encoding (function getCountryCodeDefaultMapping), while lines 112 & 113 calls a function called convertDVBUTF8.

    getCountryCodeDefaultMapping is defined in encoding.cpp line 86. You can see the mapping is taken from a config files in /etc/enigma2/encoding.conf in your STB. Lines 39, 50, 53 and 61 shows the four available formats acceptable in encoding.conf.

    convertDVBUTF8 is defined in estring.cpp at line 370. If SH follows the spec, line 423 would handle the extra 0x15 at the first, and send out the rest of the line as UTF8. Too bad, SH doesn't follow the DVB-SI spec.

    Take a moment to analyze these 3 .cpp files, and you'll see that there is no way to support SH's Chinese EITs without modifying C++ code...

    -- End of Part 2 --

  4. The Following 2 Users Say Thank You to i1646tvk84423 For This Useful Post:

    howpeter (31st March, 2014), pirsquared8 (27th April, 2013)

  5. #3
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    -- Part 3 --

    Since you need to modify code, now's a good time to start preparing your dev environment. Follow the steps @ http://users.telenet.be/spstefansp/openpliubuntu.html for a OpenPLi 2.1 dev environment. Don't try the 3.0, it has its own set of issues which I'm not ready to tackle right now.

    Similar to http://thaidreambox.bayore.net/index.php?topic=15889, you need to add
    Code:
    ("chi zho", _("Chinese")),
    between line 326 & 327 in UsageConfig.py. Now scp the new .py files to your STB:
    Code:
    scp build-dm800se/tmp/sysroots/mipsel-oe-linux/usr/lib/enigma2/python/Components/UsageConfig.py root@DM800SE_IP:/usr/lib/enigma2/python/Components/
    And delete the previous .pyo compiled version in your STB:
    Code:
    rm /usr/lib/enigma2/python/Components/UsageConfig.pyo
    Restart enigma2 GUI to take effect.
    Code:
    kill `pidof enigma2`
    Update your primary & secondary EPG language as follows:



    Because SH sends English EIT for English programmes, but both Chinese and English EIT for Chinese programmes, you'll therefore want Chinese as the primary EPG language and English as the secondary language.

    -- End of Part 3 --
    Last edited by i1646tvk84423; 17th January, 2013 at 11:04 AM.

  6. The Following 2 Users Say Thank You to i1646tvk84423 For This Useful Post:

    howpeter (31st March, 2014), pirsquared8 (5th August, 2013)

  7. #4
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    -- Part 4 --

    Add the following two lines after line 373 in estring.cpp:
    Code:
    if (table == 99)
    return std::string((char*)data, len);
    Recompile and a few enigma2 binaries will appear. Use "find" to find all of them. They should be of 2 sizes. Big 31MB ones that are not stripped and small 2.3MB ones that are stripped.

    scp one of the 2.3MB stripped ones up to your STB:
    Code:
    scp build-dm800se/tmp/work/dm800se-oe-linux/enigma2-2.7+git0+551e8f8bc26d672d0213065789239194258bd758-r36/package/usr/bin/enigma2 root@DM800SE_IP:/usr/bin/
    If scp complains file is busy, it is because your enigma2 is running. /sbin/init 4 to kill it before you scp.
    Start or restart enigma2 to take effect.

    Make your own encoding.conf:
    Code:
    cp /usr/share/enigma2/encoding.conf /etc/enigma2/
    and add one line:
    Code:
    chi ISO8859-99
    after line 9 in /etc/enigma2/encoding.conf

    -- End of Part 4 --

  8. The Following 2 Users Say Thank You to i1646tvk84423 For This Useful Post:

    howpeter (31st March, 2014), pirsquared8 (27th April, 2013)

  9. #5
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    -- Part 5 --

    Final piece of the puzzle.

    The .ttf font /usr/share/fonts/nmsbd.ttf does not have Chinese, so you need to replace its use with a .ttf font that has Chinese. A candidate is simsun.ttf from MS.

    Becareful. Your STB flash do not have much space left after OpenPLi, something like 16.9MB left. simsun.ttf is 14.6MB.
    Code:
    scp simsun.ttf root@DM800SE_IP:/usr/share/fonts/
    Then edit the skin definition:
    Code:
    cd /usr/share/enigma2/PLi-HD/
    vi skin.xml
    On line 15, change
    Code:
    <font name="Regular" filename="nmsbd.ttf" scale="100"></font>
    to
    Code:
    <font name="Regular" filename="simsun.ttf" scale="100"></font>
    Start or restart enigma2.
    Voil?!



    Side note: While the Chinese characters of simsun.ttf looks not bad, its English letters sux. Look at http://en.wikipedia.org/wiki/Unicode_fonts for alternatives.
    -- End --

  10. The Following 2 Users Say Thank You to i1646tvk84423 For This Useful Post:

    howpeter (31st March, 2014), pirsquared8 (27th April, 2013)

  11. #6
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    Names of channels, specified in lamedb, can also be in Chinese UTF8. They will show when you're using a Unicode font.

    Latest update: Microsoft's "Arial Unicode" looks the nicest, but it is more than 23MB big, way bigger than the ~ 17MB free space left after gjstroom's .nfi image. The next best looking font is "LiHei Pro", taken from my Mac OSX, which is what I've settled on. It is only 10MB, holds less glyphs than "Arial Unicode" but good enough for SH's Chinese so far..
    Last edited by i1646tvk84423; 20th January, 2013 at 09:04 AM.

  12. The Following 2 Users Say Thank You to i1646tvk84423 For This Useful Post:

    howpeter (31st March, 2014), pirsquared8 (27th April, 2013)

  13. #7
    Junior Member
    Join Date
    Dec 2011
    Posts
    22
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    6
    Thanked in
    5 Posts

    Default

    Omg, you're good.

    one question why use chi zho?

    is this hardcoded in the code for chinese?

    ("chi zho", _("Chinese")),


    And can provide the recompiled files for the estring.cpp

    Add the following two lines after line 373 in estring.cpp:
    Code: Check this link
    if (table == 99) return std::string((char*)data, len);

    i do not have a linux environment to compile this. Thanks a lot!
    Last edited by flye; 19th January, 2013 at 02:25 PM.

  14. The Following User Says Thank You to flye For This Useful Post:

    howpeter (31st March, 2014)

  15. #8
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    Quote Originally Posted by flye View Post
    one question why use chi zho?

    is this hardcoded in the code for chinese?

    ("chi zho", _("Chinese")),
    Normally, 1 language is one 3-letter code in ISO639, but some languages, including Chinese have 2. One is for simplified Chinese, the other for Traditional. The full list can be found at http://en.wikipedia.org/wiki/List_of_ISO_639-2_codes

    SH uses only "chi" from what I can see. I put both codes just to be technically correct.

    The same ISO639 codes are embedded in .mkv files that can contain multi-lang audio streams or subtitles streams.

    Quote Originally Posted by flye View Post
    And can provide the recompiled files for the estring.cpp

    i do not have a linux environment to compile this. Thanks a lot!
    1 concern is that version of your .nfi and modified engima2 binary must match. I got my .nfi latest from gjstroom ************* and my dev enironment pull down the latest source codes, so very high chance they will match.

    My .nfi is dated 2013-01-15 and now, I see the latest gjstroom .nfi is already 2013-01-20.

    I can put up my mod enigma2 binary later tonight, but as you can understand, no guarantees.

    Ideally, you should have a dev enironment. If you can create a virtual linux machine, it will be easy.
    Last edited by satsmo; 22nd January, 2013 at 11:02 PM. Reason: Removed link to forum of similar content

  16. #9
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    Quote Originally Posted by flye View Post
    And can provide the recompiled files for the estring.cpp
    My enigma2 is at https://www.dropbox.com/s/q2aiiuvu84xheus/enigma2

    My UsageConfig.py is at http://www.dropbox.com/s/qabfprzj9h3dsnu/UsageConfig.py

  17. The Following User Says Thank You to i1646tvk84423 For This Useful Post:

    pirsquared8 (16th August, 2013)

  18. #10
    Junior Member
    Join Date
    Dec 2011
    Posts
    22
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    6
    Thanked in
    5 Posts

    Default

    dear sir, did anyone ever told you that you're a genius??






    Oh man, you should have join earlier..... thanks !!!

    time to dump cnigma and use my fav openpli !!




    Last edited by flye; 22nd January, 2013 at 05:20 PM.

  19. The Following 2 Users Say Thank You to flye For This Useful Post:

    jatrophaseed (15th June, 2014), pirsquared8 (5th August, 2013)

  20. #11
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    Saw your attached pic. Well done!!
    Glad my investigation help.

  21. The Following User Says Thank You to i1646tvk84423 For This Useful Post:

    pirsquared8 (27th April, 2013)

  22. #12
    Newbie
    Join Date
    Jul 2010
    Posts
    19
    Thanks Thanks Given 
    1
    Thanks Thanks Received 
    0
    Thanked in
    0 Posts

    Default

    "My enigma2 is at https://www.dropbox.com/s/q2aiiuvu84xheus/enigma2

    My UsageConfig.py is at http://www.dropbox.com/s/qabfprzj9h3dsnu/UsageConfig.py"


    Thanks for the explanation. As a non-coder. Is there a way for me to make those changes on my STB? (DM800HD-se)

    Hopefully I can let this Chinese EPG issue be done and over with.

    Thanks
    Last edited by palmy; 25th January, 2013 at 05:42 AM.

  23. #13
    Newbie
    Join Date
    Dec 2010
    Posts
    16
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    0
    Thanked in
    0 Posts

    Default

    Dude, great job man.
    Could you get non-pli images, such as Newnigma2 to work as well?

  24. #14
    Junior Member i1646tvk84423's Avatar
    Join Date
    Jan 2013
    Location
    Singapore
    Posts
    36
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    18
    Thanked in
    11 Posts

    Default

    A major reason I showed my steps is so that others can use this knowledge and apply it to any variants of enigma2 with source code.

    As u can see above, the mod is just a 2-liner here and there. I'm 99.9% sure it will work on any variant of enigma2.

  25. #15
    Newbie
    Join Date
    Dec 2010
    Posts
    16
    Thanks Thanks Given 
    0
    Thanks Thanks Received 
    0
    Thanked in
    0 Posts

    Default

    i1646tvk84423, previously with dm800 i was able to get FULL english channel list and epg. But now with dm800se, i'm getting a couple chinese channels with garbage programme listing.

    What should I do if I want full english?
    using newnigma 3.3.2

 

 
Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
This website uses cookies
We use cookies to store session information to facilitate remembering your login information, to allow you to save website preferences, to personalise content and ads, to provide social media features and to analyse our traffic. We also share information about your use of our site with our social media, advertising and analytics partners.