Hello,
I just updated my machine with the latest OpenBSC code from git. In the past under heavy data load (EDGE), the BTS would crash. Now, after updating the code, the BTS stays up, but the SGSN crashes. Below are a few details about my setup and debugging output.
BTS: NanoBTS 165BU 1900 PC: Atom Z530 1GB ram, Running Debian (Wheezy)
Output from gdb:
<000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xc3123cCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xac6064CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xdc7cc7CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x18af79CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x84caf1CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xf1f6b6CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xd9e63bCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xbadc11CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=64 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=197 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xb7e07cCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x987e00CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xf386aaCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x38700eCMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x62743eCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x1cba00CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x6e4346CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xb20578CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x49f5d0CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x760e46CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xc839b5CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xacd5a3CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xdaf317CMD=UI DATA <0011> gprs_bssgp.c:747 BSSGP BVCI=3 Rx Flow Control BVC <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x7d7a91CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x1eef1aCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xd83b65CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xf6a775CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x855a58CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x18aa6fCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x167b76CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xce3dde6f Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x55d812CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500
Program received signal SIGFPE, Arithmetic exception. fc_queue_timer_cfg (fc=fc@entry=0x808def0) at gprs_bssgp.c:596 596 msecs = (fcqe->llc_pdu_len * 1000) / fc->bucket_leak_rate; (gdb)
OpenBSC Console Output:
Fri Jun 21 16:36:19 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Fri Jun 21 16:36:20 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:315 OC=GPRS-NSVC(f2) INST=(00,00,ff) STATE CHG: OP_STATE=Disabled AVAIL=Failed(01) Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:315 OC=GPRS-NSE(f0) INST=(00,ff,ff) STATE CHG: OP_STATE=Disabled AVAIL=Dependency(05) Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:1757 OC=GPRS-NSE(f0) INST=(00,ff,ff) Sending OPSTART Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:315 OC=GPRS-CELL(f1) INST=(00,00,ff) STATE CHG: OP_STATE=Disabled AVAIL=Dependency(05) Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:1757 OC=GPRS-CELL(f1) INST=(00,00,ff) Sending OPSTART Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:315 OC=GPRS-NSE(f0) INST=(00,ff,ff) STATE CHG: OP_STATE=Disabled AVAIL=Dependency(05) Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:315 OC=GPRS-CELL(f1) INST=(00,00,ff) STATE CHG: OP_STATE=Disabled AVAIL=Dependency(05) Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:2418 OC=GPRS-NSE(f0) INST=(00,ff,ff) IPACCESS(0xf6): SET ATTR ACK Fri Jun 21 16:38:00 2013 <0005> abis_nm.c:2418 OC=GPRS-CELL(f1) INST=(00,00,ff) IPACCESS(0xf6): SET ATTR ACK Fri Jun 21 16:38:42 2013 <0005> abis_nm.c:315 OC=GPRS-NSVC(f2) INST=(00,00,ff) Failure Event Report Type=communication failure Severity=critical failure Probable cause= 03 03 11 Additional Text=
Let me know if you need any further information.
Regards,
Caleb
On Fri, Jun 21, 2013 at 05:05:22PM -0700, Caleb Pal wrote:
<000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500
Program received signal SIGFPE, Arithmetic exception. fc_queue_timer_cfg (fc=fc@entry=0x808def0) at gprs_bssgp.c:596 596 msecs = (fcqe->llc_pdu_len * 1000) / fc->bucket_leak_rate; (gdb)
(gdb) p fc->bucket_leak_rate
It seems to be 0.
Kind regards, -Alexander Huemer
Hi Caleb,
On Fri, Jun 21, 2013 at 05:05:22PM -0700, Caleb Pal wrote:
Program received signal SIGFPE, Arithmetic exception. fc_queue_timer_cfg (fc=fc@entry=0x808def0) at gprs_bssgp.c:596 596 msecs = (fcqe->llc_pdu_len * 1000) / fc->bucket_leak_rate; (gdb)
What you should always do when reporting crashes is to include a full backtrace (bt full) as well as 'list' to show some surrounding lines of code.
However, in this case it is quite obvious. Please try the attached (untested) patch against libosmocore.
What happens is basically that the BTS wants the SGSN to stop all downlink GPRS transmission.
Harald,
Thanks for the patch to test. I applied it, and now I am back to the BTS crashing under heavy load, although it did survive a bit longer. Shortly after the BTS crashed, the sgsn crashed, and the relevant output from the OpenBSC console and gdb is below.
OpenBSC console:
Sat Jun 22 20:00:41 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=processing failure Severity=warning level failure Probable cause= 03 00 01 Additional Text=38137:WARN:GBHSS_UDPE:udp_entity.c#503:S: 736 1712 5592 1000 1416 672 888 1536 2440 848 5416 1052 816 1112 584 1488
Sat Jun 22 20:00:41 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=processing failure Severity=warning level failure Probable cause= 03 00 01 Additional Text=38140:WARN:GBHSS_UDPE:udp_entity.c#503: 496 672 424 968 784 952 1072 904 520 496 512 3176 504 888 1048 2832 1072 864
Sat Jun 22 20:00:41 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=processing failure Severity=warning level failure Probable cause= 03 00 01 Additional Text=38142:WARN:GBHSS_UDPE:udp_entity.c#503: 1168 1824 1128 496 512 424
Sat Jun 22 20:00:52 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Sat Jun 22 20:00:54 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (5 occurences)
Sat Jun 22 20:02:04 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Sat Jun 22 20:02:07 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Sat Jun 22 20:03:14 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Sat Jun 22 20:03:16 2013 <0005> abis_nm.c:315 OC=BASEBAND-TRANSCEIVER(04) INST=(00,00,ff) Failure Event Report Type=quality of service failure Severity=warning level failure Probable cause= 03 05 01 Additional Text=UDP overflow alarm on port 23000 (1 occurences)
Sgsn/gdb
<0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x717dcfCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x7e714bCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x6d27e3CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=105 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xb00b8bCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x4fcc88CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=151 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=131 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xa0115bCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xbc7635CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x690bd2CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x24fc1cCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xf06ef5CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x65d78cCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x4c0b92CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xe9acbeCMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x239284CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xf6d681CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xf4b3e6CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x92d6faCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xea0d6dCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xe78cddCMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x1dd741CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x757059CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x4d6e75CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=83 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xcb0b55CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xbb0405CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0x3c3339CMD=UI DATA <0011> gprs_bssgp.c:376 BSSGP TLLI=0xc50a3bee Rx UPLINK-UNITDATA <0012> gprs_llc.c:502 LLC SAPI=3 C FCS=0xfdcb47CMD=UI DATA <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1470 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1470 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=377 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=105 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=105 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=105 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=140 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=55 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=55 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=131 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=55 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=55 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=116 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=115 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=151 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=131 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=83 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=109 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1470 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=81 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=52 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=120 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=60 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1470 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=377 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1470 <0010> gprs_ns.c:489 NSEI=102 Tns-alive expired more then 10 times, blocking NS-VC <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=55 <0010> gprs_ns.c:573 NSEI=102 is not alive, cannot send <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=55 <0010> gprs_ns.c:573 NSEI=102 is not alive, cannot send <000f> sgsn_libgtp.c:425 GTP DATA IND from GGSN, length=1500 <0010> gprs_ns.c:573 NSEI=102 is not alive, cannot send
Program received signal SIGABRT, Aborted. 0xb7fe1424 in __kernel_vsyscall () (gdb)
(gdb) bt full #0 0xb7fe1424 in __kernel_vsyscall () No symbol table info available. #1 0xb79eb941 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #2 0xb79eed72 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #3 0xb7bb4738 in talloc_abort (reason=<optimized out>) at talloc.c:167 No locals. #4 0xb7bb8d2a in talloc_chunk_from_ptr (ptr=0x808f0b0) at talloc.c:190 pp = 0x808f0b0 "" tc = 0x808f080 #5 _talloc_free (ptr=0x808f0b0) at talloc.c:517 No locals. #6 talloc_free (ptr=ptr@entry=0x808f0b0) at talloc.c:990 No locals. #7 0xb7baeaab in msgb_free (m=m@entry=0x808f0b0) at msgb.c:72 No locals. #8 0x0804df54 in sndcp_send_ud_frag (fs=0xbfffd7cc) at gprs_sndcp.c:423 lle = 0x808e164 sne = 0x808f020 fmsg = 0x808f0b0 max_payload_len = <optimized out> len = <optimized out> rc = <optimized out> more = 1 #9 sndcp_unitdata_req (msg=msg@entry=0x809a420, lle=0x808e164, nsapi=5 '\005', mmcontext=mmcontext@entry=0x808e950) at gprs_sndcp.c:471 rc = 0 fs = {frag_nr = 1 '\001', msg = 0x809a420, next_byte = 0x809a6d4 "h=/\r\nSet-Cookie: bb_thread_lastview=bea63447ee9fe14d112a986a3c0e3923c0d88807a-17-%7Bi-3751_i- 1371319030_i-3686_i-1371306998_i-3620_i-1370565014_i-3675_i-1370754429_i-370 1_i-1371277053_i-3511_i-1369170"..., sne = 0x808f020, mmcontext = 0x808e950} #10 0x0804fbf3 in cb_data_ind (lib=lib@entry=0xb7bd1a20, packet=packet@entry=0xbfffd928, len=1500) at sgsn_libgtp.c:477 pinfo = {mode = BSSGP_PAGING_PS, scope = BSSGP_PAGING_BSS_AREA, raid = {mnc = 0, mcc = 0, lac = 0, rac = 0 '\000'}, bvci = 0, imsi = 0x0, ptmsi = 0x0, drx_params = 7680, qos = "\274\267X"} pdp = 0x808edf8 mm = 0x808e950 msg = 0x809a420 #11 0xb7bc8ea9 in gtp_gpdu_ind (gsn=gsn@entry=0x808ccb0, version=version@entry=1, peer=peer@entry=0xbfffd90c, fd=fd@entry=11, pack=pack@entry=0xbfffd91c, len=1512) at gtp.c:2714 hlen = <optimized out> pdp = 0xb7bd1a20 #12 0xb7bc957a in gtp_decaps1u (gsn=0x808ccb0) at gtp.c:3162 buffer = "2\377\005\340\000\000\000\001\017\342\000\000E \005\334\001\332@\000\062\006ÓC\347\030\020\n#\a\202\000P\350\321|\a\031\251 \335\024\233,P\020>0\357^\000\000HTTP/1.1 200 OK\r\nDate: Sun, 23 Jun 2013 03:03:48 GMT\r\nServer: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips mod_auth_passthrough/2.1 mod_bwlimited/1.4 Fr"... peer = {sin_family = 2, sin_port = 26632, sin_addr = {s_addr = 50799370}, sin_zero = "\000\000\000\000\000\000\000"} ---Type <return> to continue, or q <return> to quit--- peerlen = 16 status = <optimized out> pheader = 0xbfffd91c fd = 11 #13 0xb7bae7e2 in osmo_select_main (polling=polling@entry=0) at select.c:158 flags = <optimized out> ufd = 0x80593d4 tmp = 0x807c3f0 readset = {__fds_bits = {0 <repeats 32 times>}} writeset = {__fds_bits = {0 <repeats 32 times>}} exceptset = {__fds_bits = {0 <repeats 32 times>}} work = 1 rc = <optimized out> no_time = {tv_sec = 0, tv_usec = 0} #14 0x0804a3c1 in main (argc=1, argv=0xbffffda4) at sgsn_main.c:368 dummy_network = {country_code = 61428, network_code = 47103, name_long = 0x0, name_short = 0xbffffc04 "\f\375\377\277\004N\234\267\370\255\377\267\060\374\377\277\260\374\377\277 \016", auth_policy = 3086928783, reject_cause = 3087006400, a5_encryption = -1214510736, neci = -1212686476, send_mm_info = -1207980552, handover = {active = -1073742960, win_rxlev_avg = 0, win_rxqual_avg = 24, win_rxlev_avg_neigh = 3082283604, pwr_interval = 3080455104, pwr_hysteresis = 3628221240, max_distance = 3086973856}, stats = {chreq = {total = 0xa, no_channel = 0x0}, handover = {attempted = 0x0, no_channel = 0x1, timeout = 0xae, completed = 0xb79c0540, failed = 0xb79cd824}, loc_upd_type = {attach = 0xd696910, normal = 0xbffffbe4, periodic = 0xb7feada6, detach = 0xb79d3a30}, loc_upd_resp = {reject = 0x804946a, accept = 0xbffffcb0}, paging = {attempted = 0x8049355, detached = 0xb79c0400, completed = 0xbfff0002, expired = 0xb7ff0ae0}, sms = {submitted = 0x8049355, no_receiver = 0xb7ba8aa0, delivered = 0xb7ffeff4, rp_err_mem = 0xb79c4bf8, rp_err_other = 0x7}, call = {mo_setup = 0xbffffc64, mo_connect_ack = 0xb7feb425, mt_setup = 0x1, mt_connect = 0xb7ba8000}, chan = {rf_fail = 0xb7fea562, rll_err = 0xb7fd7000}, bts = {oml_fail = 0x7e10, rsl_fail = 0xb7ffeff4}}, mncc_state = 0xbffffd0c, mncc_recv = 0xb79c4e04, upqueue = { next = 0xb7ffadf8, prev = 0xbffffc30}, trans_list = {next = 0xbffffcb0, prev = 0xe}, bsc_api = 0xb79cd824, num_bts = 3080455120, bts_list = { next = 0xf63d4e2e, prev = 0x0}, T3101 = 10, T3103 = 0, T3105 = 0, T3107 = 1, T3109 = 2210, T3111 = -1214512128, T3113 = -1212511480, T3115 = 134517786, T3117 = -1214455724, T3119 = 134514968, T3122 = 1, T3141 = -1207963660, subscr_expire_timer = {node = {rb_parent_color = 3221224784, rb_right = 0xb7fffac0, rb_left = 0xbffffd24}, list = {next = 0xb7feb662, prev = 0xbffffd14}, timeout = {tv_sec = 134514968, tv_usec = -1073742584}, active = 0, cb = 0, data = 0xb79c0400}, rrlp = {mode = RRLP_MODE_MS_BASED}, dtx_enabled = 0, ctype_by_chreq = {GSM_LCHAN_SDCCH, 3087005960, GSM_LCHAN_SDCCH, GSM_LCHAN_SDCCH, GSM_LCHAN_NONE, GSM_LCHAN_SDCCH, 3080652277, 168972, 3082276864, GSM_LCHAN_NONE, 3221224784, 3081899232, 134519124, 3086943632, 134582576, 3221224696}, pag_any_tch = 134556299, bsc_data = 0x1, keep_subscr = -1073742428, sms_queue = 0xbffffdac, ctrl = 0xbffffcf8} rc = <optimized out> (gdb)
(gdb) list 280 281 static const struct log_info gprs_log_info = { 282 .filter_fn = gprs_log_filter_fn, 283 .cat = gprs_categories, 284 .num_cat = ARRAY_SIZE(gprs_categories), 285 }; 286 287 288 int main(int argc, char **argv) 289 { (gdb)
Let me know if you need any further information.
Regards,
Caleb
-----Original Message----- From: openbsc-bounces@lists.osmocom.org [mailto:openbsc-bounces@lists.osmocom.org] On Behalf Of Harald Welte Sent: Saturday, June 22, 2013 0051 To: Caleb Pal Cc: openbsc@lists.gnumonks.org Subject: Re: SGSN Crash Report
Hi Caleb,
On Fri, Jun 21, 2013 at 05:05:22PM -0700, Caleb Pal wrote:
Program received signal SIGFPE, Arithmetic exception. fc_queue_timer_cfg (fc=fc@entry=0x808def0) at gprs_bssgp.c:596 596 msecs = (fcqe->llc_pdu_len * 1000) / fc->bucket_leak_rate; (gdb)
What you should always do when reporting crashes is to include a full backtrace (bt full) as well as 'list' to show some surrounding lines of code.
However, in this case it is quite obvious. Please try the attached (untested) patch against libosmocore.
What happens is basically that the BTS wants the SGSN to stop all downlink GPRS transmission.
On Sat, Jun 22, 2013 at 08:17:20PM -0700, Caleb Pal wrote:
Program received signal SIGABRT, Aborted. 0xb7fe1424 in __kernel_vsyscall () (gdb)
(gdb) bt full #0 0xb7fe1424 in __kernel_vsyscall () No symbol table info available. #1 0xb79eb941 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #2 0xb79eed72 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #3 0xb7bb4738 in talloc_abort (reason=<optimized out>) at talloc.c:167 No locals. #4 0xb7bb8d2a in talloc_chunk_from_ptr (ptr=0x808f0b0) at talloc.c:190 pp = 0x808f0b0 "" tc = 0x808f080 #5 _talloc_free (ptr=0x808f0b0) at talloc.c:517 No locals. #6 talloc_free (ptr=ptr@entry=0x808f0b0) at talloc.c:990 No locals. #7 0xb7baeaab in msgb_free (m=m@entry=0x808f0b0) at msgb.c:72 No locals. #8 0x0804df54 in sndcp_send_ud_frag (fs=0xbfffd7cc) at gprs_sndcp.c:423
This looks like the known double free. My workaround/solution has been described here[1]. Could you please generate a PCAP file, I will then look in what it takes to re-produce the issue..
thanks holger
[1] http://lists.osmocom.org/pipermail/openbsc/2013-March/004492.html
-
Alexander Huemer -
Caleb Pal -
Harald Welte -
Holger Hans Peter Freyther