new sanitizer breakage: SIGSEGV in sgsn_create_pdp_ctx()
The sanitizer build used to get through to testing the PCU, now it already fails at openbsc's sgsn test. This happens in the recently added test_pdp_deactivation_with_pdp_ctx: http://jenkins.osmocom.org/jenkins/job/Osmocom_Sanitizer/388/consoleFull commit 1611df5226199da2bf2fba3d22d93cc1a6c6c777 Commit: Pravin Kumarvel <pmanohar@radisys.com> CommitDate: Mon Dec 12 17:20:39 2016 +0530 Support Deactivate PDP Context Request from network https://gerrit.osmocom.org/1262 I can reproduce the segmentation fault locally, but only when the sanitizer is enabled. When stepping up to the failure and checking the parameters, all seems to be in order; immediately when trying to step into sgsn_create_pdp_ctx(), the SIGSEGV is fired. So far the actual failure is not clear to me, I haven't found the 0x02 pointer yet that asan complains about: ==21897==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000002 I found a use-after-free which isn't the cause for above asan failure: gsm0408_gprs_access_cancelled(mm, GMM_CAUSE_GPRS_NOTALLOWED); LOGMMCTXP(LOGL_NOTICE, mm, "No PDP context to deactivate\n"); gsm0408_gprs_access_cancelled() calls mm_ctx_cleanup_free(), and after that the local mm is non-NULL but freed. Change the order to: LOGMMCTXP(LOGL_NOTICE, mm, "No PDP context to deactivate\n"); gsm0408_gprs_access_cancelled(mm, GMM_CAUSE_GPRS_NOTALLOWED); (This second issue is shown when removing test_pdp_deactivation_with_pdp_ctx() from test_pdp_deactivation()) The cause for the asan failure shown above and in jenkins still evades me. But I'm afraid we have to revert the patch. Please run the asan build on this patch and re-submit when the cause is clear. How to asan build has been discussed recently: http://lists.osmocom.org/pipermail/openbsc/2016-November/009901.html ~N -- - Neels Hofmeyr <nhofmeyr@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschäftsführer / Managing Directors: Harald Welte
I think this situation will repeat itself over and over again until we make sanitizer tests part of our jenkins setup. On 13.12.2016 13:08, Neels Hofmeyr wrote:
The sanitizer build used to get through to testing the PCU, now it already fails at openbsc's sgsn test. This happens in the recently added test_pdp_deactivation_with_pdp_ctx:
http://jenkins.osmocom.org/jenkins/job/Osmocom_Sanitizer/388/consoleFull
commit 1611df5226199da2bf2fba3d22d93cc1a6c6c777 Commit: Pravin Kumarvel <pmanohar@radisys.com> CommitDate: Mon Dec 12 17:20:39 2016 +0530
Support Deactivate PDP Context Request from network
https://gerrit.osmocom.org/1262
I can reproduce the segmentation fault locally, but only when the sanitizer is enabled. When stepping up to the failure and checking the parameters, all seems to be in order; immediately when trying to step into sgsn_create_pdp_ctx(), the SIGSEGV is fired. So far the actual failure is not clear to me, I haven't found the 0x02 pointer yet that asan complains about:
==21897==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000002
I found a use-after-free which isn't the cause for above asan failure:
gsm0408_gprs_access_cancelled(mm, GMM_CAUSE_GPRS_NOTALLOWED); LOGMMCTXP(LOGL_NOTICE, mm, "No PDP context to deactivate\n");
gsm0408_gprs_access_cancelled() calls mm_ctx_cleanup_free(), and after that the local mm is non-NULL but freed. Change the order to:
LOGMMCTXP(LOGL_NOTICE, mm, "No PDP context to deactivate\n"); gsm0408_gprs_access_cancelled(mm, GMM_CAUSE_GPRS_NOTALLOWED);
(This second issue is shown when removing test_pdp_deactivation_with_pdp_ctx() from test_pdp_deactivation())
The cause for the asan failure shown above and in jenkins still evades me. But I'm afraid we have to revert the patch. Please run the asan build on this patch and re-submit when the cause is clear.
How to asan build has been discussed recently: http://lists.osmocom.org/pipermail/openbsc/2016-November/009901.html
~N
-- Max Suraev <msuraev@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte
On Tue, Dec 13, 2016 at 02:42:00PM +0100, Max wrote:
I think this situation will repeat itself over and over again until we make sanitizer tests part of our jenkins setup.
That's why I'm continuously pressing for a clean sanitizer build so that we can finally add it everywhere. And I can use your help, as I said before. The openbsc already builds fine with asan now. So: Max, could you take the libosmocore asan patch a23817622b28cb1969a73ffd36da501eb29b9cd7 and apply the same to openbsc and the various libs openbsc depends on? I won't get around to it anytime soon, and I would appreciate sharing the work load on this. It should be mostly copy-paste work... I will gladly +2, and once we have those in, I can easily add asan builds to the gerrit build jobs. Thanks! ~N -- - Neels Hofmeyr <nhofmeyr@sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschäftsführer / Managing Directors: Harald Welte
participants (2)
-
Max -
Neels Hofmeyr