Hi all,
I'd like to share my odd? interesting? unnerving? experience today. All I wanted to do is quickly check something (gerrit #713) on sysmobts, but I faced a series of problems.
First some sysmobts related problems, later two problems entirely unrelated to osmocom...
(*) My sysmobts SDK no longer works: libosmo-abis now requires libortp >= 0.22.0, but the SDK I have used so far doesn't have that: poky-eglibc-i686-meta-toolchain-osmo-armv5te-toolchain-osmo-1.5.3-20160323112523.sh
(I didn't go through it in this sequence, but tried these things:)
a) tried to build a more recent libortp into the installed SDK. It goes fine up to a point. It takes a bit of work to find and fetch all the repositories that are required: git://git.linphone.org/bcunit.git git://git.linphone.org/bctoolbox.git git://git.linphone.org/ortp.git but sadly, bctoolbox wants libpolarssl aka mbedtls git@github.com:ARMmbed/mbedtls.git -- which has no automake/autoconf, only a plain Makefile! It also has cmake and others, but so far I only know how to tell the ./configure script to cross-compile and install into the SDK, so I kind of gave up on figuring out cmake for that, for now.
b) just reverted the libortp version bumping commit in libosmo-abis (15d9b7929d449e4138bcb003c614035bceadc3d1) Everything compiles and works out, but alas, I get constant very weird distortion/blip-tones/hickups on the voice channels, a bit of voice still audible but definitely some bad codec problem. A regression? Is it related to libortp? This happens both for a TS config of TCH/H and TCH/F.
c) tried fetching a new SDK from 201310-testing/sdk: poky-eglibc-i686-meta-toolchain-osmo-armv5te-toolchain-osmo-1.5.3-20160910200558.sh This has the same problem with too old libortp. Again I revert the libortp version bump from libosmo-abis. Same results as above: badly broken voice.
d) tried a new 1.5.4 SDK from 201310-nightly/sdk: poky-eglibc-i686-meta-toolchain-osmo-armv5te-toolchain-osmo-1.5.4-20161004010024.sh Here libosmo-abis is fine with the libortp it finds, so far so good. Sadly, this doesn't even do hickuped voice, only low noise on both ends. So I decided to go back to the SDK that worked for me last time to bisect. I will post another mail in case I found a regression.
(*) For half a minute there I wondered why GPRS wasn't working until I remembered our two-phase-access aka WAIT_ASSIGN problem in osmo-pcu and reverted f1a7b8fc. With that, GPRS works fine with all of the above. phew ;)
(*) Entirely unrelated to osmocom: I launch my CN programs from a script that launches multiple terminals. I happen to use rxvt-unicode for that and always have for a year. Turns out this time urxvt strips the LD_LIBRARY_PATH when it is invoked. I still don't know how this changed, but when I drop the suid that I find for unknown reasons sitting on the urxvt binary's group ('chmod g-s /usr/bin/urxvt'), it lets the LD_LIBRARY_PATH be, and the binaries run. How did this change from the last time I ran my CN scripts?? No idea.
(*) Same with sudo: LD_LIBRARY_PATH now all of a sudden gets stripped during my sudo ggsn -f -c ggsn.conf invocation. I use sudo for ggsn to allow establishing tunnels. Again I didn't change anything about my scripts, just today it doesn't find the libraries in /usr/local/lib anymore. I have to manually add it like sudo LD_LIBRARY_PATH=/usr/local/lib ggsn ... but then sudo tells me I'm not allowed to set the env. After searching around, I add 'SETENV:' to my sudoers file: neels ALL= NOPASSWD:SETENV: /usr/local/bin/ggsn * This time ggsn starts again like it always used to. How did this change from last time I ran ggsn this way?? No idea at all.
So my quick test took a bit longer than expected (hasn't started), and the night has progressed at an alarming rate... I'll write again when I found anything on the broken voice.
~Neels
On Wed, Oct 05, 2016 at 03:43:02AM +0200, Neels Hofmeyr wrote:
b) just reverted the libortp version bumping commit in libosmo-abis (15d9b7929d449e4138bcb003c614035bceadc3d1) Everything compiles and works out, but alas, I get constant very weird distortion/blip-tones/hickups on the voice channels, a bit of voice still audible but definitely some bad codec problem. A regression?
I have identified voice to be broken on sysmobts-v2 starting from this osmo-bts commit:
commit 654175f33bd412671e3ef8cdd65c0689d10f278c Refs: 0.4.0-281-g654175f Author: Max msuraev@sysmocom.de AuthorDate: Sat Sep 17 14:15:03 2016 +0200 Commit: Max msuraev@sysmocom.de CommitDate: Sat Sep 24 15:09:31 2016 +0200
DTX: check Marker bit to send ONSET to L1
If Marker bit is set than it's a talkspurt which we have to explicitly indicate to L1 by first sending ONSET message and than actual voice data in a separate message.
This change affect sysmobts and LC15 hw.
Change-Id: I88c41568bcb0d82699f617adc4ad192603dd1bb6 Related: OS#1750
Going back with osmo-bts to one before this commit fixes all problems with both 1.5.3 SDK versions tested:
poky-eglibc-i686-meta-toolchain-osmo-armv5te-toolchain-osmo-1.5.3-20160323112523.sh poky-eglibc-i686-meta-toolchain-osmo-armv5te-toolchain-osmo-1.5.3-20160910200558.sh
However, the 1.5.4 SDK still produces only noise. It has a newer libortp, and note that the litecell15 instructions include a choice of libortp 0.16.5, explicitly *not* a newer version, so it might be related to that.
To summarize, to have a working setup on sysmobts, I still need to: - use a 1.5.3 SDK - libosmo-abis: git revert 15d9b7929d449e4138bcb003c614035bceadc3d1 - osmo-bts: git checkout eb08a87be52bf55a78c12689f1e5c7cff627cbbc
GPRS is not affected, works well with all of the above (except for the old two-phase-access OS#1756 issue).
Maybe I'll figure out how to compile the mbedtls into the 1.5.3 SDK or build without mbedtls, allowing a newer libortp: if voice is reduced to noise by that, we would know whether libortp's HEAD has a problem for us.
~Neels
Hi.
This might be related to commit c77c2a6aa13accbc558888ab788d1148eb9aeb1a in osmobts - see https://projects.osmocom.org/issues/1661 If you roll-back this commit, does voice re-appear?
Also, why do you have to use 1.5.3 instead of 1.5.4?
On 10/05/2016 04:49 AM, Neels Hofmeyr wrote:
However, the 1.5.4 SDK still produces only noise. It has a newer libortp, and note that the litecell15 instructions include a choice of libortp 0.16.5, explicitly *not* a newer version, so it might be related to that.
[SDK 1.5.4 has no voice]
This might be related to commit c77c2a6aa13accbc558888ab788d1148eb9aeb1a in osmobts - see https://projects.osmocom.org/issues/1661
osmo-bts has no commit c77c2a6aa13accbc558888ab788d1148eb9aeb1a. ...you mean libosmo-abis.
If you roll-back this commit, does voice re-appear?
Wow, it actually does! I was pretty sure this would not fix anything. I have discussed in detail in OS#1661, and see https://gerrit.osmocom.org/1011 .
A pressing issue which you seemingly ignored is that your osmo-bts commit 654175f33bd412671e3ef8cdd65c0689d10f278c breaks voice on SysmoBTS. Could you grab a sysmoBTS and verify+fix it soon? Otherwise revert.
~Neels
Hi.
I have not ignored anything, I've pointed to commit which I suspect could have broken voice (as you rightfully pointed out it's in libosmo-abis). Could you clarify if reverting this commit in libosmo-abis fixes the voice for sysmobts master? Or in addition to reverting c77c2a6aa13accbc558888ab788d1148eb9aeb1a in libosmo-abis I've got to checkout some later commit in sysmobts?
If reverting commit in libosmo-abis indeed fixes voice for sysmobts sdk 1.5.4 than please send patch for review (add me as reviewer).
Right now I'm working on DTX rewrite using FSM which might fix sysmobts if it's indeed broken (after fixing libosmo-abis).
On 10/06/2016 04:01 PM, Neels Hofmeyr wrote:
[SDK 1.5.4 has no voice]
This might be related to commit c77c2a6aa13accbc558888ab788d1148eb9aeb1a in osmobts - see https://projects.osmocom.org/issues/1661
osmo-bts has no commit c77c2a6aa13accbc558888ab788d1148eb9aeb1a. ...you mean libosmo-abis.
If you roll-back this commit, does voice re-appear?
Wow, it actually does! I was pretty sure this would not fix anything. I have discussed in detail in OS#1661, and see https://gerrit.osmocom.org/1011 .
A pressing issue which you seemingly ignored is that your osmo-bts commit 654175f33bd412671e3ef8cdd65c0689d10f278c breaks voice on SysmoBTS. Could you grab a sysmoBTS and verify+fix it soon? Otherwise revert.
~Neels
On Thu, Oct 06, 2016 at 04:09:18PM +0200, Max wrote:
Hi.
I have not ignored anything, I've pointed to commit which I suspect could have broken voice (as you rightfully pointed out it's in libosmo-abis). Could you clarify if reverting this commit in libosmo-abis fixes the voice for sysmobts master? Or in addition to reverting c77c2a6aa13accbc558888ab788d1148eb9aeb1a in libosmo-abis I've got to checkout some later commit in sysmobts?
If reverting commit in libosmo-abis indeed fixes voice for sysmobts sdk 1.5.4 than please send patch for review (add me as reviewer).
Right now I'm working on DTX rewrite using FSM which might fix sysmobts if it's indeed broken (after fixing libosmo-abis).
These are two distinct issues. The one with no voice at all happens on 1.5.4 and is fixed by gerrit #1011.
The other issue is that SysmoBTS voice is hickuped/distorted starting from your first DTX commit 654175f33bd412671e3ef8cdd65c0689d10f278c on osmo-bts. (This on both 1.5.3 and 1.5.4, with and without #1011 applied.)
So please grab a SysmoBTS and test DTX against it :)
~Neels
On Thu, Oct 06, 2016 at 08:20:03PM +0200, Neels Hofmeyr wrote:
I have not ignored anything, I've pointed to commit which I suspect could have broken voice (as you rightfully pointed out it's in libosmo-abis). Could you clarify if reverting this commit in libosmo-abis fixes the voice for sysmobts master? Or in addition to reverting c77c2a6aa13accbc558888ab788d1148eb9aeb1a in libosmo-abis I've got to checkout some later commit in sysmobts?
If reverting commit in libosmo-abis indeed fixes voice for sysmobts sdk 1.5.4 than please send patch for review (add me as reviewer).
These are two distinct issues. The one with no voice at all happens on 1.5.4 and is fixed by gerrit #1011.
Can you please clarify. When you say "on 1.5.4" you mean that you used a SDK/toolchain from 201310-nightly. However, did you also run a 201310-nightly filesystem on the BTS at that time?
The recent changes in libosmo-abis require the use of a more recent libortp. The filesystem + SDK of 201310-nightly contains that later version and should thus be expected to work.
What is of course not supposed to work is a filesystem (including ortp binary) of 201310-stable or -testing with a osmo-bts binary built on a SDK/toolchain from 201310-nightly.
Please confirm that this is not the case in your testing.
Regards, Harald
On Fri, Oct 07, 2016 at 08:10:29PM +0200, Harald Welte wrote:
Can you please clarify. When you say "on 1.5.4" you mean that you used a SDK/toolchain from 201310-nightly. However, did you also run a 201310-nightly filesystem on the BTS at that time?
So, what I usually do is
osmo* ---make-install--> SDK-sysroot ---copy--> bts:~/neels/lib bts: export LD_LIBRARY_PATH=~/neels/lib ; ~/neels/bin/osmo-bts
In words, compile the entire osmo toolchain that I use from libosmocore up, installing *into* my computer's SDK's sysroot, and when all is built I copy the binaries I want to test as well as the entire usr/lib/ from the SDK over to the BTS ($SYSMO_SDK/sysroots/armv5te-poky-linux-gnueabi/usr/lib). In the environment to run those built binaries on the BTS, I point LD_LIBRARY_PATH at the location where I copied the SDK's libraries. This should run the binaries in the library scenario as intended by the SDK, also including the osmo libs built in the same cycle, not actually using the older libs in bts:/usr/lib.
That's a quick way around re-installing the BTS from scratch. That should be fine, right? Should I rather overwrite the entire root fs?
Thanks,
~Neels
-
Harald Welte -
Max -
Neels Hofmeyr