Hi.
While trying to debug issue with osmo-pcu failing to allocate tfi due to pdch unavailability I've noticed that regular connection attempts are also failing with imm. ass. reject due to channel unavailability.
I do not see rach flood so it feels like misconfiguration or maybe hw issue.
MY question is - how to troubleshoot this? Where can I look as to which channels are available, which are occupied and by whom?
thanks, Max.
On Sat, Nov 14, 2015 at 01:29:37PM +0100, Suraev wrote:
MY question is - how to troubleshoot this? Where can I look as to which channels are available, which are occupied and by whom?
'show lchan' shows you the state of currently allocated logical channels. 'by whom' doesn't really work unless we have received an identity of the subscirber already. You don't know at the time of RACH request.
14.11.2015 21:20, Harald Welte пишет:
On Sat, Nov 14, 2015 at 01:29:37PM +0100, Suraev wrote:
MY question is - how to troubleshoot this? Where can I look as to which channels are available, which are occupied and by whom?
'show lchan' shows you the state of currently allocated logical channels.
That's what have puzzled me most - "show lchan" returns nothing but "imm. ass. reject" is still the constant reply.
'by whom' doesn't really work unless we have received an identity of the subscirber already. You don't know at the time of RACH request.
Anyway, it turned out that the problem was between the chair and the keyboard: I've used wrong branch of Osmo-BTS which was incompatible with my hw.
Now everything seems to be fine component wise - all logs seems adequate, calls going through.
The only issue - no internet on the device, even name resolution fails. Seems like packets are not forwarded properly. What would the best place to poke into user's traffic - see what packets come from which phone and what happens to them and to return traffic?
cheers, Max.
On Wed, Nov 18, 2015 at 09:43:10PM +0100, Suraev wrote:
Anyway, it turned out that the problem was between the chair and the keyboard: I've used wrong branch of Osmo-BTS which was incompatible with my hw.
which branch is the incompatible branch, and which branch are you using now? We have merged osmo-trx support into master in order to get rid of of the need of the nightmare of too many branches. It would be a pity if all that effort was for nothing.
The only issue - no internet on the device, even name resolution fails. Seems like packets are not forwarded properly. What would the best place to poke into user's traffic - see what packets come from which phone and what happens to them and to return traffic?
does the GPRS control plane work, i.e. do you se GPRS ATTACH / ACCEPT, ROUTING AREA UPDATE ATTACH/ ACCEPT, PDP CONTEXT ACTIVATE / ACCEPT?
You can follow this in the VTY logging of OsmoSGSN (logging level for mobility management and session management).
Alternatively, the Gb interface on port 23000/udp can be followed n wireshark. You need to 'decode as... Gb (or was it called GPRS-Gb?)
19.11.2015 09:26, Harald Welte пишет:
which branch is the incompatible branch, and which branch are you using now? We have merged osmo-trx support into master in order to get rid of of the need of the nightmare of too many branches. It would be a pity if all that effort was for nothing.
Hmm, I thought that process was still ongoing. I'm using branch 201509-fairwaves-rebase of Osmo-BTS with usrp b210. The rest of the components are at master branch.
does the GPRS control plane work, i.e. do you se GPRS ATTACH / ACCEPT, ROUTING AREA UPDATE ATTACH/ ACCEPT, PDP CONTEXT ACTIVATE / ACCEPT?
You can follow this in the VTY logging of OsmoSGSN (logging level for mobility management and session management).
You mean: telnet localhost 4245 en logging enable logging set-log-mask DMM:DCC ?
Alternatively, the Gb interface on port 23000/udp can be followed n wireshark. You need to 'decode as... Gb (or was it called GPRS-Gb?)
I can decode it as GPRS (wireshark 1.12.7). In this case I've got:
gprs attach request, with id request followed by attach accept and attach complete.
The tun0 is created, masquerading is on by iptables, forwarding is enabled with sysctl. After the gprs attach I see lots of FLOW-CONTROL_BVC and NS-ALIVE messages with corresponding -ACK.
The strange thing is - I see no packets over tun0 at all. Same goes for udp port 23000 except what was mentioned above. Shouldn't I be able to at least see dns queries from mobile?
The logs from OsmoPCU also looks cryptic but the message about timeout seems suspicious:
<0001> sysmo_sock.cpp:283 PCU-SYSMO socket has been connected <0001> pcu_l1_if.cpp:343 BTS available <0008> gprs_ns.c:233 NSVCI=65534 Creating NS-VC <0008> gprs_ns.c:233 NSVCI=101 Creating NS-VC <0008> gprs_ns.c:1561 NSEI=101 RESET procedure based on API request <0008> gprs_ns.c:392 NSEI=101 Tx NS RESET (NSVCI=101, cause=O&M intervention) <0001> pcu_l1_if.cpp:80 Sending activate request: trx=0 ts=6 <0001> pcu_l1_if.cpp:469 PDCH: trx=0 ts=6 <0001> pcu_l1_if.cpp:80 Sending activate request: trx=0 ts=7 <0001> pcu_l1_if.cpp:469 PDCH: trx=0 ts=7 <0008> gprs_ns.c:935 NSVCI=101 Rx NS RESET ACK (NSEI=101, NSVCI=101) <0008> gprs_ns.c:501 NSEI=101 Tx NS UNBLOCK (NSVCI=101) <0008> gprs_ns.c:1345 NSEI=101 Rx NS UNBLOCK ACK <000a> gprs_bssgp_pcu.cpp:489 NS-VC 101 is unblocked. <0009> gprs_bssgp_pcu.cpp:749 Sending reset on BVCI 0 <0009> gprs_bssgp_bss.c:290 BSSGP (BVCI=0) Tx BVC-RESET CAUSE=8 <0009> gprs_bssgp_pcu.cpp:757 Sending reset on BVCI 2 <0009> gprs_bssgp_bss.c:290 BSSGP (BVCI=2) Tx BVC-RESET CAUSE=8 <0009> gprs_bssgp_pcu.cpp:765 Sending unblock on BVCI 2 <0009> gprs_bssgp_bss.c:270 BSSGP (BVCI=2) Tx BVC-BLOCK <0001> pcu_l1_if.cpp:295 RACH request received: sapi=1 qta=0, ra=120, fn=1742448 <0009> tbf_ul.cpp:407 LLC [PCU -> SGSN] TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FLOW) len=52 <0009> gprs_bssgp_pcu.cpp:179 LLC [SGSN -> PCU] = TLLI: 0x865ad706 IMSI: 000 len: 9 <0007> gprs_rlcmac_meas.cpp:104 UL RSSI of TLLI=0x865ad706: -62 dBm <0001> pcu_l1_if.cpp:295 RACH request received: sapi=1 qta=0, ra=123, fn=1742930 <0007> gprs_rlcmac_meas.cpp:159 DL packet loss of IMSI=000 / TLLI=0x865ad706: 0% <0009> tbf_ul.cpp:407 LLC [PCU -> SGSN] TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FLOW) len=17 <0009> gprs_bssgp_pcu.cpp:179 LLC [SGSN -> PCU] = TLLI: 0x865ad706 IMSI: 000 len: 9 <0007> gprs_rlcmac_meas.cpp:104 UL RSSI of TLLI=0x865ad706: -66 dBm <0002> tbf.cpp:434 TBF(TFI=0 TLLI=0x865ad706 DIR=DL STATE=ASSIGN) poll timeout for FN=1743252 (curr FN 1743313) <0002> tbf.cpp:485 - Timeout for polling PACKET CONTROL ACK for PACKET DOWNLINK ASSIGNMENT. <0002> tbf.cpp:777 - Assignment was on PACCH <0002> tbf.cpp:785 - No downlink ACK received yet <0002> bts.cpp:812 Recovered downlink assignment for TBF(TFI=0 TLLI=0x865ad706 DIR=DL STATE=FLOW) <0007> gprs_rlcmac_meas.cpp:184 DL Bandwitdh of IMSI=000 / TLLI=0x865ad706: 0 KBits/s <0007> gprs_rlcmac_meas.cpp:159 DL packet loss of IMSI=000 / TLLI=0x865ad706: 0% <0009> tbf_ul.cpp:407 LLC [PCU -> SGSN] TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FLOW) len=17 <0009> gprs_bssgp_pcu.cpp:179 LLC [SGSN -> PCU] = TLLI: 0x865ad706 IMSI: 901708776992222 len: 26 <0002> tbf.cpp:434 TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FINISHED) poll timeout for FN=1743534 (curr FN 1743599) <0002> tbf.cpp:485 - Timeout for polling PACKET CONTROL ACK for PACKET DOWNLINK ASSIGNMENT. <0002> tbf.cpp:777 - Assignment was on PACCH <0002> tbf.cpp:785 - No downlink ACK received yet <0002> bts.cpp:812 Recovered downlink assignment for TBF(TFI=0 TLLI=0x865ad706 DIR=DL STATE=FLOW) <0007> gprs_rlcmac_meas.cpp:184 DL Bandwitdh of IMSI=000 / TLLI=0x865ad706: 0 KBits/s <0007> gprs_rlcmac_meas.cpp:159 DL packet loss of IMSI=000 / TLLI=0x865ad706: 0% <0009> tbf_ul.cpp:407 LLC [PCU -> SGSN] TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FLOW) len=17 <0009> gprs_bssgp_pcu.cpp:179 LLC [SGSN -> PCU] = TLLI: 0x865ad706 IMSI: 901708776992222 len: 26 <0002> tbf.cpp:434 TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FINISHED) poll timeout for FN=1743534 (curr FN 1743599) <0002> tbf.cpp:485 - Timeout for polling PACKET CONTROL ACK for PACKET DOWNLINK ASSIGNMENT. <0002> tbf.cpp:777 - Assignment was on PACCH <0002> tbf.cpp:779 - Uplink data was received <0002> tbf.cpp:434 TBF(TFI=0 TLLI=0x865ad706 DIR=UL STATE=FINISHED) poll timeout for FN=1743638 (curr FN 1743703) <0007> gprs_rlcmac_meas.cpp:184 DL Bandwitdh of IMSI=901708776992222 / TLLI=0x865ad706: 0 KBits/s <0007> gprs_rlcmac_meas.cpp:104 UL RSSI of TLLI=0x865ad706: -67 dBm <0007> gprs_rlcmac_meas.cpp:159 DL packet loss of IMSI=901708776992222 / TLLI=0x865ad706: 0% <0001> pcu_l1_if.cpp:295 RACH request received: sapi=1 qta=0, ra=121, fn=1744126 <0009> tbf_ul.cpp:407 LLC [PCU -> SGSN] TBF(TFI=0 TLLI=0xc7c9a645 DIR=UL STATE=FLOW) len=8 <0007> gprs_rlcmac_meas.cpp:104 UL RSSI of TLLI=0xc7c9a645: -69 dBm <0002> bts.cpp:767 PACKET CONTROL ACK with unknown FN=1745900 TLLI=0xc7c9a645 (TRX 0 TS 6) <0002> bts.cpp:767 PACKET CONTROL ACK with unknown FN=1746095 TLLI=0xc7c9a645 (TRX 0 TS 6)
Any ideas what am I missing in here?
cheers, Max.
Hi Max,
On Thu, Nov 19, 2015 at 12:41:42PM +0100, Suraev wrote:
Hmm, I thought that process was still ongoing.
from my point of view, l1sap and osmo-trx have been merged.
Any additional osmo-trx specific fixes that might be missing must be submitted by those people who are maintaining that port (and actually using it).
Unfortunately I have not yet seen any such patches submitted to the mailing list.
logging enable logging set-log-mask DMM:DCC
no:
logging enable logging filter all 1 logging level mm debug logging level sm debug
gprs attach request, with id request followed by attach accept and attach complete.
If you're not getting a pdp context activate, then you phone is not activating a pdp context. Without pdp context, no user data.
After the gprs attach I see lots of FLOW-CONTROL_BVC and NS-ALIVE messages with corresponding -ACK.
That's just signalling between PCU and SGSN, unrelate to the MS.
The strange thing is - I see no packets over tun0 at all. Same goes for udp port 23000 except what was mentioned above. Shouldn't I be able to at least see dns queries from mobile?
The MS is not activating a pdp context. Befoer that you will not see any data. Check your apn configuration.
20.11.2015 11:12, Harald Welte пишет:
logging level sm debug
I don't see this one but logging level lgtp debug and logging level gprs debug did the trick, thanks!
If you're not getting a pdp context activate, then you phone is not activating a pdp context. Without pdp context, no user data.
Doh! As far as I recall sgsn/ggsn ignores apn (btw, there's nothing while searching for "apn" in wiki) but I forgot that phone still got to have something set as apn ("internet" works fine) in order to start actually using gprs.
The MS is not activating a pdp context. Befoer that you will not see any data. Check your apn configuration.
I should have guessed - the lack of gprs indicator on the phone means it did not actually attempt to use it.
Now, it works, although just one way for some reason.
I can see activated pdp context in sgsn vty:
OsmoSGSN# show pdp-context all PDP Context IMSI: 901708776992222, SAPI: 3, NSAPI: 5 APN: internet PDP Address: IPv4 192.168.0.2 SGSN PDP Context Statistics: User Data Messages ( In): 16 (0/s 0/m 16/h 0/d) User Data Messages (Out): 0 (0/s 0/m 0/h 0/d) User Data Bytes ( In): 1032 (0/s 0/m 1032/h 0/d) User Data Bytes (Out): 0 (0/s 0/m 0/h 0/d)
Also I see the dns requests in tcpdump over tun0 interface.
The problem is the above 0 Out - nothing has been sent to the mobile.
The masquerading on the host seems to work fine at first glance: ping -S 192.168.0.3 8.8.8.8 gets expected response (here 192.168.0.3 is the address from the space assigned to phones).
However, none of the phone's requests receive reply. On the other hand if I try to ping phone directly from the host than I got reply just fine, counters got updated in sgsn, messages are visible in wireshark etc.
So it seems like the osmocom side of things is doing fine and the remaining issue is somewhere on the host. I would blame iptables but I've checked that no blocking rules installed and all policies defaults to accept for both regular and nat tables.
Anyway, will keep digging, thanks for help!
cheers, Max.
On Sun, Nov 22, 2015 at 05:20:18PM +0100, Suraev wrote:
Doh! As far as I recall sgsn/ggsn ignores apn (btw, there's nothing while searching for "apn" in wiki) but I forgot that phone still got to have something set as apn ("internet" works fine) in order to start actually using gprs.
The APN may be used to find the proper GGSN to establish a PDP context with (APN and parts of the IMSI are used to compose a "*.gprs" name, which is then resolved using DNS to find the GGSN's IP address). If your SGSN is configured to send things to a specific GGSN, the APN does in fact not have an impact. So it depends on your SGSN's configuration.
I can see activated pdp context in sgsn vty:
OsmoSGSN# show pdp-context all PDP Context IMSI: 901708776992222, SAPI: 3, NSAPI: 5 APN: internet PDP Address: IPv4 192.168.0.2 SGSN PDP Context Statistics: User Data Messages ( In): 16 (0/s 0/m 16/h 0/d) User Data Messages (Out): 0 (0/s 0/m 0/h 0/d) User Data Bytes ( In): 1032 (0/s 0/m 1032/h 0/d) User Data Bytes (Out): 0 (0/s 0/m 0/h 0/d)
Also I see the dns requests in tcpdump over tun0 interface.
The problem is the above 0 Out - nothing has been sent to the mobile.
I was at first unsure because an SGSN receives and sends both to/from mobile as well as to/from GGSN. But looking at the code, it seems that your interpretation is correct.
The masquerading on the host seems to work fine at first glance: ping -S 192.168.0.3 8.8.8.8 gets expected response (here 192.168.0.3 is the address from the space assigned to phones).
Could you explain the -S? In my host system's man page, -S is documented as:
-S sndbuf Set socket sndbuf. If not specified, it is selected to buffer not more than one packet.
and busybox's ping doesn't document any -S option ... ? So to me it seems your ping command sets a send buffer of 192 packets (ignoring ".168.0.3") and pings google's nameserver.
I'm still new on the subject, just hoping to be helpful... I've recently done GTP and PDP context related stuff, so a few shots in the dark:
In wireshark, you could examine which GSN addresses the SGSN is telling the GGSN to use ("GPRS Tunneling Protocol" / "GSN address"). There's two, the first is for the Ctrl plane, the second for User. The GGSN will reply to the PDP context creation and then switch to those GSN addresses for subsequent messages. If those are wrong/not reachable from the GGSN side, that may cause lost replies?
Have you enabled forwarding on the host running the GGSN? I'm using these commands to be able to locally test with a "fake" GGSN on my laptop: sudo -s echo 1 > /proc/sys/net/ipv4/ip_forward iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
~Neels
-
Harald Welte -
Neels Hofmeyr -
Suraev