osmo-gsm-tester: Issue with osmo-gsm-tester-runner Jenkins job
Hi all, We're trying to setup osmo-gsm-tester on bare metal and are currently experiencing issues with the Jenkins job 'osmo-gsm-tester-runner'. All of our test machines are running Ubuntu Server 20.04 and have been setup following the latest available manual from https://downloads.osmocom.org/docs/latest/osmo-gsm-tester-manual.pdf. We've trying to use the '4g_srsLTE' example from https://github.com/osmocom/osmo-gsm-tester/tree/master/doc/examples/4g_srsLT... but when trying to run the 'ping' test the following error is returned in the Jenkins console output: " 14:07:52.867022 run mk-remote-dir(pid=2480): ERR: Terminated: ERROR {rc=1} [trial-21↪4g:srsenb-rftype@soapy↪ping.py:9↪ping.py↪srsepc_10.100.100.113↪host-jenkins@10.100.100.113↪mk-remote-dir(pid=2480)] 14:07:53.015620 run mk-remote-dir(pid=2480): stdout: | (launched: 2022-09-23_14:07:51.598612) | mkdir: cannot create directory ‘/osmo-gsm-tester-srsepc’: Permission denied " It appears that osmo-gsm-tester is trying to create the directory 'osmo-gsm-tester-srsepc' under root, which is not permitted as the Jenkins user the script runs has does not have permissions. Additionally, it looks like this path is hard-coded in the Python module https://github.com/osmocom/osmo-gsm-tester/blob/master/src/osmo_gsm_tester/o... and also the same applies for srsENB https://github.com/osmocom/osmo-gsm-tester/blob/master/src/osmo_gsm_tester/o... Is this the intended behaviour? Or have we missed out steps during our bare-metal setup process? I've double-checked that the main unit can SSH into the slave unit without a password so that's not the issue causing "permission denied". Best regards, Callum.
Hi Callum, Yes that path is hardcoded in the code, and you need to have that path manually created and properly chown'ed before using it with osmo-gsm-tester. See https://gitea.osmocom.org/osmocom/docker-playground/src/branch/master/osmo-g... on the steps required to set it up. You can also have a look a osmo-gsm-tester User Manual for info in general too: https://ftp.osmocom.org/docs/latest/osmo-gsm-tester-manual.pdf Regards, Pau -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Pau, Thanks for your advice. We're now seeing srsepc being deployed to the machine and attempting to run but the test job eventually fails as a timeout expires after five minutes. It appears as if srsepc is being told to run without any configuration as per the output of "ps aux" below: jenkins 10292 0.0 0.0 2608 596 pts/0 Ss+ 16:07 0:00 sh -c sudo osmo-gsm-tester_setcap_net_admin.sh /osmo-gsm-tester-srsepc/srslte/bin/srsepc root 10293 0.0 0.0 8816 4388 pts/0 S+ 16:07 0:00 sudo osmo-gsm-tester_setcap_net_admin.sh /osmo-gsm-tester-srsepc/srslte/bin/srsepc The full job log is available here as a GitHub Gist: https://gist.github.com/callumsnowden/8c1c52353f8adbdead318612d4f9efe4 There looks to be a full configuration successfully generated and I can see this on the slave machine. Our test suite configuration is located here https://github.com/callumsnowden/osmo-gsm-tester/tree/lc_srsran_config_testi... and only has changes relating to the hardware configuration for the test suite. Is there some other configuration that needs to be done to get srsepc to launch with the generated configuration? Any help/advice you could offer would be most appreciated. Best, Callum.
Hi Callum, the process you are seeing running is not "srspec", but the helper script "osmo-gsm-tester_setcap_net_admin.sh" run with sudo in order to set the required capabilitites to the "srsepc" binary before actually running it. So to me it looks like the job is stalled running "sudo" which will probably be asking for a password. I bet if you check the output log of "setcap-netadm" process under the osmo-gsm-tester output directory you will find sudo asking for a password printed there. You probably need to configure sudo on that machine to avoid requesting a password for that specific script (yes, it may pose a security risk). See osmo-gsm-tester.git "utils/bin/osmo-gsm-tester_setcap_*" and "utils/sudoers.d/osmo-gsm-tester_setcap_*", and check osmo-gsm-tester manual for further info. regards, Pau -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Pau, Once again many thanks, we've successfully deployed srsEPC, srsENB and had a USB modem connect successfully. The last step that is causing issues is running the "ping" command in the example. On the console we're seeing an error of "10:16:42.774663 run ping(pid=1427): Launched 10:16:43.976875 run ping(pid=1427): ERR: Terminated: ERROR {rc=255} [trial-69↪4g:srsenb-rftype@lime↪ping.py:23↪ping.py↪/sierra_0↪ping(pid=1427)] 10:16:44.112929 run ping(pid=1427): stdout: | (launched: 2022-09-30_10:16:42.708515) | 1428: waiting for 1429 | child exited with 255 10:16:44.247787 run ping(pid=1427): stderr: | (launched: 2022-09-30_10:16:42.754778) | Cannot open network namespace "1-1.2.2": No such file or directory" On the main machine all the scripts are in place ("osmo-gsm-tester_netns_exec.sh", "osmo-gsm-tester_netns_setup.sh") with appropriate permissions so that our Jenkins user can execute the scripts with sudo. The ping test we're running is here https://github.com/callumsnowden/osmo-gsm-tester/blob/lc_srsran_config_testi... Have you got any advice that would make this final step work? Best, Callum.
Hi Callum, First of all, IIRC nobody has tried to run ofono modems against the SRS/open5gs ojects in osmo-gsm-tester (this doesn't mean it's not possible, just that nobody tried that yet AFAIC). The ofono modems have been used usually against the 2g osmocom stack objects. For 4G the users were more interested in using srsUE or amrisoft UE (either physically through USRP or virtually). Hence you may have to fix some stuff or slightly change the 4g tests to get everything working perfectly (the MS objects try to be generic, independent from its implementation, but may not be the case 100%). As per the issue you reported: I don't remember the details now (It's been a while), but I think you may need to create the network namespaces yourself manually after the host startup and before running osmo-gsm-tester. Check osmo-gsm-tester.git helper "utils/modem-netns-setup.py" to create them. You may also want to check this presentation which talks about related topics (network namespace setup in osmo-gsm-tester): https://media.ccc.de/v/osmodevcon2019-119-osmo-gsm-tester-e-gprs-test-setup Regards, Pau -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Pao, Thanks for your reply. I did try running the "modem-netns-setup.py" script which makes the modems invisible to ofono as expected. Deleting the namespaces then restarting the ofono service makes them reappear. I've watched your talk and read the accompanying slide deck, and you mentioned a patch being put in place that works around this removal issue - do you have details of this patch so we can apply it and try? This removal issue also seems to be talked about in https://osmocom.org/issues/3881 when modems crash, which happens frequently. https://github.com/osmocom/osmo-gsm-tester/blob/b172c066fb73e0156f9be2f024ae... also seems to deal with moving modem interfaces into network namespaces, and following this example https://github.com/osmocom/osmo-gsm-tester/blob/master/sysmocom/suites/gprs/... I tried adding the calls to "activate_context" and "setup_context_data_plane" to our 4G ping test script, which crashed out with an error "GDBus.Error:org.ofono.Error.NotAttached: GPRS is not attached (36)". Best, Callum.
Hi Callum, For use with osmo-gsm-tester we keep our own downstream fork at sysmocom: https://gitea.sysmocom.de/sysmocom/ofono This repo contains an "osmo-gsm-tester" branch based on upstream's master (a bit outdated now, may need some rebasing work): https://gitea.sysmocom.de/sysmocom/ofono/src/branch/osmo-gsm-tester That branch contains mostly QMI fixes and improvements (like initial call support) which hasn't yet been submitted/merged in upstream. On top of those fixes and improvements there are a few patches named "NOT-FOR-MERGE" or "NOT-FOR-UPSTREAM", which contain some handy modifications for our own osmo-gsm-tester, which are not of interest or cannot be merged upstream. The "NOT_FOR_MERGE: decode qmi messages with libqmi" patch will help you in debugging whatever ofono is doing in QMI with the modem. The patch "NOT-FOR-MERGE: ignore disconnecting network interface" is the patch you are looking for. Note that ofono versions others than the branch I presented are not known to work properly with osmo-gsm-tester. Regards, Pau -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Pau, We've now tried the "osmo-gsm-tester" branch in your Git repository but struggled to get it to build. We followed the steps documented in the Ansible "gsm-tester" https://github.com/osmocom/osmo-ci/blob/master/ansible/roles/gsm-tester/task... role which clones the repository, switches branch and then begins config/compilation. On a Debian 9 machine the error we're seeing consists of the line "make[1]: *** No rule to make target 'ell/util.c', needed by 'ell/util.lo'. Stop.". The same error also occurs on a Debian 10 system, and our intended Ubuntu 20.04 machine. Are we missing a step that needs to be performed first? Best, Callum.
Hi, you seem to be missing the Embedded Linux library (ell) in there. See the same foono.git/README file for more info it's explained there. AFAICT the one we are using here is ell.git b8d44f9a70cceb8544452cc1390b895f7bd2b786 (release 0.27). Regards, Pau -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
Hi Pau, Many thanks for all your help, we eventually got everything working. We did originally try cloning the ELL repository linked in the ofono readme and placing it next to the ofono folder, but this didn't work. However your pointer to the specific version of ELL (0.27) did work, so thanks for that. We did look through the Ansible role that sets up osmo-gsm-tester but couldn't see anywhere that ELL is cloned, so not quite sure how you're setting that up? Regardless, we got everything working so once again thanks for your help. Best, Callum
Hi Callum, On 10/11/22 16:22, Callum Snowden wrote:
We did look through the Ansible role that sets up osmo-gsm-tester but couldn't see anywhere that ELL is cloned, so not quite sure how you're setting that up?
Maybe we did that step manually or had to do it after we upgraded/rebased to a newer version of ofono. We have 1 testing and 1 prod setup so it's not like we are deploying everything frequently. Feel free to submit patches to improve the ansible recipes. Regards, Pau -- - Pau Espin Pedrol <pespin@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
participants (3)
-
Callum Snowden -
callum＠abopen.com
-
Pau Espin Pedrol