Hi list,
Attached 3 patches for decoding UCS2 MT-SMS and incoming USSD. Main idea: convert UCS2 text to UTF-8, check if terminal supports UTF-8 and if yes - print, otherwise - print in hex, and notify VTY to check mobile log (since we can't send UTF text to VTY, if someone knows how to do it - please tell me).
Waiting for comments.
Thanks, Pavel
Pavel Baturko wrote:
Attached 3 patches for decoding UCS2 MT-SMS and incoming USSD. Main idea: convert UCS2 text to UTF-8, check if terminal supports UTF-8 and if yes - print, otherwise - print in hex, and notify VTY to check mobile log (since we can't send UTF text to VTY, if someone knows how to do it - please tell me).
hi paval,
the patches for osmocombb look good to me, except that the UTF encoded text is not only displayed on VTY, but also not written to sms.txt file. look at gsm340_rx_sms_deliver() where linefeeds from sms-text are removed for the vty output. you could also remove (or replace by '?') non-ascii characters there. then it can be displayed on VTY too.
you can apply the osmocombb patches, after the libosmocore patch is applied to master of libosmocore. for applying the libosmocore patch, i cannot give green light, because holger was recently working on sms code, so i would ask him.
regards,
andreas
Hi Andreas,
Thanks for comments.
the patches for osmocombb look good to me, except that the UTF encoded text is not only displayed on VTY, but also not written to sms.txt file. you could also remove (or replace by '?') non-ascii characters there. then it can be displayed on VTY too.
Am I right that you are proposing to display only ascii part of text and save it also in sms.txt file instead of status messages like 'SMS was printed ...'? For all non-ascii SMSs this will looks like '??? ?? ?????' at VTY, is it ok?
you can apply the osmocombb patches, after the libosmocore patch is applied to master of libosmocore. for applying the libosmocore patch, i cannot give green light, because holger was recently working on sms code, so i would ask him.
I'll wait for Holger's answer.
Thanks, Pavel
Pavel Baturko wrote:
Am I right that you are proposing to display only ascii part of text and save it also in sms.txt file instead of status messages like 'SMS was printed ...'? For all non-ascii SMSs this will looks like '??? ?? ?????' at VTY, is it ok?
i don't know about coding of UTF, but wikipedia reads that all ASCII characters are UTF-8 characters, so i guess that all characters except non-ASCII character should be visible. am i wrong?
Hi,
i don't know about coding of UTF, but wikipedia reads that all ASCII characters are UTF-8 characters, so i guess that all characters except non-ASCII character should be visible. am i wrong?
Yes, they will be visible, but I'm talking about SMSs without ASCII chars except spaces/numbers/dots/commas/etc. In that case all words (in national language for example) will not be visible at VTY and user will see many '?' symbols if change non-ascii chars to '?' and spaces/comms/etc, and I think such representation of original text will be not very good (if I correctly understand your first comment).
Thanks, Pavel
Yes, they will be visible, but I'm talking about SMSs without ASCII chars except spaces/numbers/dots/commas/etc. In that case all words (in national language for example) will not be visible at VTY and user will see many '?' symbols if change non-ascii chars to '?' and spaces/comms/etc, and I think such representation of original text will be not very good (if I correctly understand your first comment).
ok then, why not print your message about encoded type to VTY, but store message in sms.txt. the gsm340_rx_sms_deliver() function could take two arguments, one text for the vty and one to store.
Hi,
Attached new version. SMS text will be saved to sms.txt file in any case (in case of UCS2 SMSs with non-ASCII chars this file should be opened with UTF-8 encoding) but on VTY user will see notice that SMS text can't be printed.
Thanks, Pavel
Hi,
Attached new version. SMS text will be saved to sms.txt file in any case (in case of UCS2 SMSs with non-ASCII chars this file should be opened with UTF-8 encoding) but on VTY user will see notice that SMS text can't be printed.
Thanks, Pavel
hi pavael,
this looks good to me now. you can commit it as soon as the libosmocore patch is accepted and committed.
best regards,
andreas
this looks good to me now. you can commit it as soon as the libosmocore patch is accepted and committed.
Should I ask someone for committing to libosmocore? And can I commit to osmocom-bb repo directly (I think I can't)? Or I should re-ask for committing when libosmocore will be ready?
Thanks, Pavel
Hi Pavel,
sorry to be the party pooper, but I don't like the libosmocore patch for two reasons:
1) a popen to the locale command is really ugly. First of all, it will synchronously block the process issuing that command. Typical libosmocore-using processes are single-threaded and expect library calls to be non-blocking. Secondly, I really hate the idea of spawning other processes, executing other executables. It sounds lieke an ugly hack. Is there no way to get the information programmatically, e.g. by using localeconv(3) ?
2) the function "ucs2_to_utf8_decode" is cluttering the global namespace even further. Yes, it is in principle not a function related to gsm, but I'd still appreciate if it would start with osmo_ or gsm_ prefix in order to make sure we don't clash with applications or other libraries using the same symbol name.
Hi Harald,
Is there no way to get the information programmatically, e.g. by using localeconv(3) ?
Another way to get this information that I know is smthng like that: + old_locale = setlocale(LC_ALL, NULL); + curr_locale = setlocale(LC_ALL, ""); + for (i = 0; i < strlen(curr_locale); i++) + curr_locale[i] = tolower(curr_locale[i]); + have_utf8_locale = ((strstr(curr_locale, "utf-8") != NULL) || + (strstr(curr_locale, "utf8" ) != NULL)) ? 1 : 0; + setlocale(LC_ALL, old_locale); Not sure if it's ok because of temporary changing of current locale. unfortunately I was not able to find better ways to check locale.
- the function "ucs2_to_utf8_decode" is cluttering the global namespace even further. Yes, it is in principle not a function related to gsm, but I'd still appreciate if it would start with osmo_ or gsm_ prefix in order to make sure we don't clash with applications or other libraries using the same symbol name.
I can move it to utils.c, is it acceptable? And of course add "osmo_" prefix.
Thanks, Pavel
Hi Pavel,
On Fri, Feb 08, 2013 at 01:55:52PM +0400, Pavel Baturko wrote:
Another way to get this information that I know is smthng like that:
old_locale = setlocale(LC_ALL, NULL);curr_locale = setlocale(LC_ALL, "");for (i = 0; i < strlen(curr_locale); i++)curr_locale[i] = tolower(curr_locale[i]);have_utf8_locale = ((strstr(curr_locale, "utf-8") != NULL) ||(strstr(curr_locale, "utf8" ) != NULL)) ? 1 : 0;setlocale(LC_ALL, old_locale);Not sure if it's ok because of temporary changing of current locale. unfortunately I was not able to find better ways to check locale.
the locale setting is only for the current program. And as indicated, at least so far, most libosmocore users are single-threaded, i.e. nobody would notice if the locale was different for a very short amount of time. Furthermore, assuming that the code is not called all that often (i.e. only once at program startup) I wouldn't be worried about that so much.
I am not an expert in locale.h matters, but it would somehow be strange to me if you cannot determine utf-8 support without having to actually change the locale. Is anyone on this list more familiar with it?
- the function "ucs2_to_utf8_decode" is cluttering the global namespace even further. Yes, it is in principle not a function related to gsm, but I'd still appreciate if it would start with osmo_ or gsm_ prefix in order to make sure we don't clash with applications or other libraries using the same symbol name.
I can move it to utils.c, is it acceptable? And of course add "osmo_" prefix.
yes, that would be appreciated, thanks.
Hi Harald,
the locale setting is only for the current program. And as indicated, at least so far, most libosmocore users are single-threaded, i.e. nobody would notice if the locale was different for a very short amount of time. Furthermore, assuming that the code is not called all that often (i.e. only once at program startup) I wouldn't be worried about that so much.
I can add some kind of global var which will indicate UTF-8 support and init it when app starting with code above (or other code if will find better one). Then use it in case of SMS/USSD receving.
I am not an expert in locale.h matters, but it would somehow be strange to me if you cannot determine utf-8 support without having to actually change the locale. Is anyone on this list more familiar with it?
I asked some time ago, but there was no answer.. setlocale function with NULL argument always returns "C" - Minimal "C" locale - if locale was not changed before, with arguments rather than NULL - changes the locale.
Thanks, Pavel
Hi Pavel,
On Fri, Feb 08, 2013 at 02:29:17PM +0400, Pavel Baturko wrote:
time. Furthermore, assuming that the code is not called all that often (i.e. only once at program startup) I wouldn't be worried about that so much.
I can add some kind of global var which will indicate UTF-8 support and init it when app starting with code above (or other code if will find better one). Then use it in case of SMS/USSD receving.
This sounds reasonable. Please make the global variable static and hide it to the caller. So only on the first call to the function it will do the setlocale() trick, then store the result in a variable. All further calls to the function will then retunr the variable instead of setlocale() and friends.
Hi,
Here are updated versions of patches: Functions for converting ucs2->utf8 and checking current locale are moved to util.c. Function for checking current locale rewrited to use 'setlocale' instead of external 'locale' cmd call. Support of UTF is checked just once in 'l23_app_init' and then used via static var.
Thanks, Pavel
Hi Harald,
Have you had a chance to look at last versions of patches?
Thanks, Pavel
On Sun, Feb 10, 2013 at 11:55 AM, Pavel Baturko pabftk@gmail.com wrote:
Hi,
Here are updated versions of patches: Functions for converting ucs2->utf8 and checking current locale are moved to util.c. Function for checking current locale rewrited to use 'setlocale' instead of external 'locale' cmd call. Support of UTF is checked just once in 'l23_app_init' and then used via static var.
Thanks, Pavel
Just kindly reminder..
Thanks, Pavel
On Wed, Feb 13, 2013 at 2:30 PM, Pavel Baturko pabftk@gmail.com wrote:
Hi Harald,
Have you had a chance to look at last versions of patches?
Thanks, Pavel
On Sun, Feb 10, 2013 at 11:55 AM, Pavel Baturko pabftk@gmail.com wrote:
Hi,
Here are updated versions of patches: Functions for converting ucs2->utf8 and checking current locale are moved to util.c. Function for checking current locale rewrited to use 'setlocale' instead of external 'locale' cmd call. Support of UTF is checked just once in 'l23_app_init' and then used via static var.
Thanks, Pavel
baseband-devel@lists.osmocom.org