Hi All,
I’m facing an issue with osmo-nitb (HLR/sqlite part) - segmentation fault. All works ok for the first usage od DB (e.g. Location Updating Request) – but there is segmentation fault on second Location Updating request or any other attempt to execute any command using VTY for HLR.
Syslog____________________________________________________________________________
Oct 12 20:32:29 debian kernel: [ 2397.242984] osmo-nitb[985]: segfault at 64692e74 ip b77c15c9 sp bf8688b0 error 4 in libdbdsqlite3.so[b77be000+8000]
Part of bt__________________________________________________________________________
#0 0xb77c15c9 in ?? () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
#1 0xb77c17e4 in ?? () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
#2 0xb77c26f7 in dbd_query () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
No symbol table info available.
Can anybody please advise?
Thank you,
Regards,
Marcin
Attached files
-Tcpump (between nanoBTS(.34) and BSC (.20))
-Debug output, backtrace, syslog below
Debug-output 1/2________________________________________________________________________
<0004> abis_rsl.c:1465 (bts=0,trx=0,ts=0,ss=0) Activating ARFCN(514) SS(0) lctype SDCCH r=LOCATION_UPDATE ra=0x01 ta=0
<0004> abis_rsl.c:1199 (bts=0,trx=0,ts=0,ss=0) CHANNEL ACTIVATE ACK
<0000> abis_rsl.c:1653 (bts=0,trx=0,ts=0,ss=0) SAPI=0 ESTABLISH INDICATION
<0000> gsm_04_08.c:3584 Dispatching 04.08 message, pdisc=5
<0002> gsm_04_08.c:1136 LOCATION UPDATING REQUEST: MI(IMSI)=260060059332155 type=NORMAL
<0001> gsm_04_08.c:145 (bts 0 trx 0 ts 0 pd 05) Sending 0x18 to MS.
<0000> abis_rsl.c:1653 (bts=0,trx=0,ts=0,ss=0) SAPI=0 DATA INDICATION
<0003> bsc_api.c:500 CLASSMARK CHANGE CM2(len=3) CM3(len=7)
<0000> abis_rsl.c:1653 (bts=0,trx=0,ts=0,ss=0) SAPI=0 DATA INDICATION
<0000> gsm_04_08.c:3584 Dispatching 04.08 message, pdisc=5
<0002> gsm_04_08.c:523 IDENTITY RESPONSE: MI(IMEI)=359809011036930
<0002> gsm_04_08.c:568 Location Updating Request procedure timedout.
<0002> gsm_04_08.c:452 Subscriber 260060059332155: LOCATION UPDATING REJECT LAC=1 BTS=0
<0001> gsm_04_08.c:145 (bts 0 trx 0 ts 0 pd 05) Sending 0x04 to MS.
<0000> chan_alloc.c:324 (bts=0,trx=0,ts=0,ss=0) starting release sequence
<0003> gsm_04_08_utils.c:239 Sending Channel Release: Chan: Number: 0 Type: 1
<0004> abis_rsl.c:616 (bts=0,trx=0,ts=0,ss=0) DEACTivate SACCH CMD
DB-ouput___________________________________________________________________________
sqlite> select * from subscriber ;
1|2015-10-12 18:31:38|2015-10-12 18:31:39|260060059332155||22685|0||0|
Debug-output 2/2_____________________________________________________________________
<0004> abis_rsl.c:1465 (bts=0,trx=0,ts=0,ss=0) Activating ARFCN(514) SS(0) lctype SDCCH r=LOCATION_UPDATE ra=0x07 ta=0
<0004> abis_rsl.c:1199 (bts=0,trx=0,ts=0,ss=0) CHANNEL ACTIVATE ACK
0000> abis_rsl.c:1653 (bts=0,trx=0,ts=0,ss=0) SAPI=0 ESTABLISH INDICATION
<0000> gsm_04_08.c:3584 Dispatching 04.08 message, pdisc=5
<0002> gsm_04_08.c:1136 LOCATION UPDATING REQUEST: MI(IMSI)=260060059332155 type=NORMAL
<0001> gsm_04_08.c:145 (bts 0 trx 0 ts 0 pd 05) Sending 0x18 to MS.
Segmentation fault (core dumped)
Syslog____________________________________________________________________________
Oct 12 20:32:29 debian kernel: [ 2397.242984] osmo-nitb[985]: segfault at 64692e74 ip b77c15c9 sp bf8688b0 error 4 in libdbdsqlite3.so[b77be000+8000]
Backtrace__________________________________________________________________________
root@debian:~/openbsc/openbsc/src/osmo-nitb# gdb osmo-nitb core
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from osmo-nitb...done.
[New LWP 985]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `./osmo-nitb --debug=DRLL:DCC:DMM:DRR:DRSL:DNM'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0xb77c15c9 in ?? () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
(gdb) bt full
#0 0xb77c15c9 in ?? () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
No symbol table info available.
#1 0xb77c17e4 in ?? () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
No symbol table info available.
#2 0xb77c26f7 in dbd_query () from /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
No symbol table info available.
#3 0xb77a7122 in dbi_conn_queryf () from /usr/lib/i386-linux-gnu/libdbi.so.1
No symbol table info available.
#4 0x0806f99f in get_equipment_by_subscr (subscr=0x9055b60) at db.c:550
result = 0x9053520
cm2 = 0xbf8688e4 ""
cm3 = 0xbf8688e4 ""
#5 0x08070523 in db_get_subscriber (field=151346016, id=0xbf868e00 "260060059332155") at db.c:897
result = 0x9053520
quoted = 0x904fa58 "x4q\267x5\005\t2015-10-12 18:31:39"
subscr = 0x9055b60
#6 0x0807423a in get_subscriber (ident=0xbf868e00 "260060059332155", type=0, sgrp=0x9010840) at gsm_subscriber.c:67
subscr = <optimized out>
#7 subscr_get_by_imsi (sgrp=0x9010840, imsi=0xbf868e00 "260060059332155") at gsm_subscriber.c:240
subscr = 0x80c20e0 <subscriber_create_cmd+32>
#8 0x0807f8a5 in mm_rx_loc_upd_req (msg=<optimized out>, conn=0x9063740) at gsm_04_08.c:643
subscr = 0x0
bts = 0xb7737b01 <osmo_timer_schedule+81>
---Type <return> to continue, or q <return> to quit---
mi_type = <optimized out>
gh = 0x9063793
mi_string = "260060059332155\000\267zs\267\230+\005\t@7 \006\t\206\353\a\b"
#9 gsm0408_rcv_mm (msg=0x8085488, conn=0x9063740) at gsm_04_08.c:1137
gh = <optimized out>
rc = 0
#10 gsm0408_dispatch (conn=0x9052b98, msg=0x9063740) at gsm_04_08.c:3594
gh = <optimized out>
pdisc = <optimized out>
rc = 0
#11 0x0807895a in msc_compl_l3 (conn=0x9052b98, msg=0x9063740, chosen_channel=0) at osmo_msc.c:49
No locals.
#12 0x0804e468 in gsm0408_rcvmsg (msg=0x9063740, link_id=0 '\000') at bsc_api.c:684
rc = -1081702172
#13 0x0806619b in abis_rsl_rx_rll (msg=<optimized out>) at abis_rsl.c:1674
ts_name = 0xbf8688e4 ""
sapi = 0 '\000'
#14 abis_rsl_rcvmsg (msg=0x9063740) at abis_rsl.c:2085
rc = 151402378
#15 0xb771f107 in handle_ts1_read (bfd=<optimized out>) at input/ipaccess.c:271
link = 0xbf8688e4
hh = 0x5
msg = 0x9063740
---Type <return> to continue, or q <return> to quit---
ts_nr = 3077720460
#16 ipaccess_fd_cb (bfd=0x9056b30, what=1) at input/ipaccess.c:386
rc = 151348668
#17 0xb77380d8 in osmo_select_main (polling=0) at select.c:160
flags = 1
ufd = 0x9056b30
tmp = 0xb7754524 <osmo_fds>
readset = {__fds_bits = {0 <repeats 32 times>}}
writeset = {__fds_bits = {0 <repeats 32 times>}}
exceptset = {__fds_bits = {0 <repeats 32 times>}}
work = 1
rc = <optimized out>
no_time = {tv_sec = 0, tv_usec = 0}
#18 0x0804c8bf in main (argc=2, argv=0xbf869214) at bsc_hack.c:356
rc = <optimized out>
Hi Marcin,
just general guessing ... since the segfault is in libdbdsqlite3, maybe it's a building/linking issue? Maybe there are multiple instances of a library, or dependencies have upgraded and it needs a clean build...?
~Neels
Hi Neels,
Thank you for your email.
There is only one instance of libdbsqlite3.so - and it's a fresh build.
root@debian:~/openbsc/openbsc/src/osmo-nitb# locate libdbdsqlite3.so /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
root@debian:~/openbsc/openbsc/src/osmo-nitb# file /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd673630ac267d5ef3f2d82665ffc5d0b80d24fc, stripped
root@debian:~/openbsc/openbsc/src/osmo-nitb# ls -ls /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so 36 -rw-r--r-- 1 root root 35696 Apr 4 2014 /usr/lib/i386-linux-gnu/dbd/libdbdsqlite3.so
Regards, Marcin
On 14 October 2015 at 23:00, Neels Hofmeyr nhofmeyr@sysmocom.de wrote:
Hi Marcin,
just general guessing ... since the segfault is in libdbdsqlite3, maybe it's a building/linking issue? Maybe there are multiple instances of a library, or dependencies have upgraded and it needs a clean build...?
~Neels
On 13 Oct 2015, at 20:12, Marcin Starzyk marcin.starzyk@gmail.com wrote:
Hi All,
I’m facing an issue with osmo-nitb (HLR/sqlite part) - segmentation fault. All works ok for the first usage od DB (e.g. Location Updating Request) – but there is segmentation fault on second Location Updating request or any other attempt to execute any command using VTY for HLR.
please use valgrind to build with the address sanitizer. There are known out og bounds reads of libdbd but you don't seem to hit those.
holger
Hi Holger,
Thank you very much for your help
I've run osmo-nitb with valgrind (trace attached). There are some errors but it seems valgrind is suppressing segmentation fault - so LUP works , which is good enough for me now. But is there any permanent solution I could work on (unfortunately I don't' understand what do you mean by using valgrind to build with the address sanitizer)?
Regards, Marcin
On 15 October 2015 at 09:30, Holger Freyther holger@freyther.de wrote:
On 13 Oct 2015, at 20:12, Marcin Starzyk marcin.starzyk@gmail.com
wrote:
Hi All,
I’m facing an issue with osmo-nitb (HLR/sqlite part) - segmentation
fault. All works ok for the first usage od DB (e.g. Location Updating Request) – but there is segmentation fault on second Location Updating request or any other attempt to execute any command using VTY for HLR.
please use valgrind to build with the address sanitizer. There are known out og bounds reads of libdbd but you don't seem to hit those.
holger
On 16 Oct 2015, at 23:40, Marcin Starzyk marcin.starzyk@gmail.com wrote:
Hi Holger,
Thank you very much for your help
I've run osmo-nitb with valgrind (trace attached). There are some errors but it seems valgrind is suppressing segmentation fault - so LUP works , which is good enough for me now. But is there any permanent solution I could work on (unfortunately I don't' understand what do you mean by using valgrind to build with the address sanitizer)?
libdi/libdbd has various memory issues. Some I fixed, some are open. In general we want to stop using it. Would you be interested in contributing a patch to use sqlite3 directly and creating a migration path (e.g. to convert the blob format used by dbd/dbi to native support in sqlite)?
holger
-
Holger Freyther -
Marcin Starzyk -
Neels Hofmeyr