Hi!
It seems we're loosing serial characters on their way from the (Linux) PC to the phone.
I think I've seen this at some point in the past, but now I'm seeing it constantly.
Let's assume we're trying to send the following sequence of bytes from the PC side to the phone in one of the HLDC connections:
send_to_phone(dlci=5): 0a 00 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
what we actually get from the sercomm callback on the phone is:
l1a_l23_rx_cb: 0a 00 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
as you can see, there is a '00 01' missing between the '00' and the '02'.
Interestingly, putting a usleep(10000) after every serial write does not solve the problem, so it is not timing related. We don't get any UART overflows on the calypso, either.
If I change the data pattern, then the first bytes are transmitted fine, but as soon as there is one 00 bytes, two characters are missing from the stream.
If I strace osmocon, it is clear that we write every character, including the null-bytes, and all of the writes are successfully.
I've again looked at (and played with) the termios settings, but could not find any problem (nor solution).
It's also not the sercomm layer that is loosing bytes, as I the characters appear already lost when printing them from within the calypso UART driver, as you can see here:
getchar_nb(1) = 7e // flag character getchar_nb(1) = 05 // hdlc getchar_nb(1) = 03 // hdlc getchar_nb(1) = 0a // l1ctl_hdr (undefined msg_type 10) getchar_nb(1) = 00 // padding getchar_nb(1) = 02 // here should be 00, 01, 02... getchar_nb(1) = 03 getchar_nb(1) = 04 getchar_nb(1) = 05 [...]
Any ideas? It's really strange and I don't really know what else to do. I'm quite sure our binaries contain multiple successive null-bytes and their download is working great...
Regards, Harald
On Sun, Mar 21, 2010 at 10:35, Harald Welte laforge@gnumonks.org wrote:
Hi!
It seems we're loosing serial characters on their way from the (Linux) PC to the phone.
I think I've seen this at some point in the past, but now I'm seeing it constantly.
Let's assume we're trying to send the following sequence of bytes from the PC side to the phone in one of the HLDC connections:
send_to_phone(dlci=5): 0a 00 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
what we actually get from the sercomm callback on the phone is:
l1a_l23_rx_cb: 0a 00 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
as you can see, there is a '00 01' missing between the '00' and the '02'.
Interestingly, putting a usleep(10000) after every serial write does not solve the problem, so it is not timing related. We don't get any UART overflows on the calypso, either.
If I change the data pattern, then the first bytes are transmitted fine, but as soon as there is one 00 bytes, two characters are missing from the stream.
If I strace osmocon, it is clear that we write every character, including the null-bytes, and all of the writes are successfully.
I've again looked at (and played with) the termios settings, but could not find any problem (nor solution).
It's also not the sercomm layer that is loosing bytes, as I the characters appear already lost when printing them from within the calypso UART driver, as you can see here:
getchar_nb(1) = 7e // flag character getchar_nb(1) = 05 // hdlc getchar_nb(1) = 03 // hdlc getchar_nb(1) = 0a // l1ctl_hdr (undefined msg_type 10) getchar_nb(1) = 00 // padding getchar_nb(1) = 02 // here should be 00, 01, 02... getchar_nb(1) = 03 getchar_nb(1) = 04 getchar_nb(1) = 05 [...]
Any ideas? It's really strange and I don't really know what else to do. I'm quite sure our binaries contain multiple successive null-bytes and their download is working great...
I had similar problems when developing an app in C that communicated with a GPS. I lost incoming bytes now and then (probably the same pattern as you see). Because of this and other reasons I switched to Python and pySerial which worked perfectly fine. I did never understand what happened either, I also tried to get as raw mode as possible.
/Erik
Regards, Harald --
- Harald Welte laforge@gnumonks.org http://laforge.gnumonks.org/
============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
On Sun, Mar 21, 2010 at 11:04:53AM +0100, Erik Ekman wrote:
Any ideas? It's really strange and I don't really know what else to do. I'm quite sure our binaries contain multiple successive null-bytes and their download is working great...
I had similar problems when developing an app in C that communicated with a GPS. I lost incoming bytes now and then (probably the same pattern as you see). Because of this and other reasons I switched to Python and pySerial which worked perfectly fine. I did never understand what happened either, I also tried to get as raw mode as possible.
Have you ever bothered to strace the python process in order to determine which termios settings it uses on the serial port? after all, the python jit will use the same system calls like any C program.
Regards, Harald
Just a follow-up:
I've crossed-over the RxD and TxD wires of my T191 cable.
The two consecutive 00 00 bytes are definitely transmitted from the PC, and the come back into the PC. The problem must be inside the phone itself.
Regards, Harald
Hi Harald,
send_to_phone(dlci=5): 0a 00 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d
...
l1a_l23_rx_cb: 0a 00 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12
as you can see, there is a '00 01' missing between the '00' and the '02'.
Does the 16C750 do anything fancy to compensate for slow/fast clocks by adjusting it's internal sampling points to the edges on the incoming data?
If he does, we might have a clock that's slightly off and the compensation might fix that for anything except the 0-bytes which don't have any 0/1 transitions.
To find out, one could use a lower baud-rate and adjust the divider until the problem gets worse/goes away. And always check with a 2nd com-port on the PC in parallel.
Chris (just lurking, no compatible phone (yet))
Just another intermediate update:
1) I've pushed my current debugging code to the uart-debug branch
Simply run osmocon + layer23 and the hello_world.bin image. layer23 will send a 62 byte sized HLDC message every 10 seconds to the phone, and various instances of the code will print what they see. The payload of the packet is supposed to be 0a 00 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05
but what the firmware sees is: 0a 00 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05
So as you can see, every '00 01' sequence has been chopped out, despite 10ms delays between the characters. If you change the payload to not include zero-bytes, all bytes are received fine.
2) Enabling various misc UART interrupt sources and checking for error conditions in LSR does not help
Regards, Harald
Maybe 0x0 is a hardware processed escape sequence. In that case you'd have to find the sequence that escapes the escape. Just a idea. -- K
2010/3/21 Harald Welte laforge@gnumonks.org
Just another intermediate update:
- I've pushed my current debugging code to the uart-debug branch
Simply run osmocon + layer23 and the hello_world.bin image. layer23 will send a 62 byte sized HLDC message every 10 seconds to the phone, and various instances of the code will print what they see. The payload of the packet is supposed to be 0a 00 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05
but what the firmware sees is: 0a 00 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05
So as you can see, every '00 01' sequence has been chopped out, despite 10ms delays between the characters. If you change the payload to not include zero-bytes, all bytes are received fine.
- Enabling various misc UART interrupt sources and checking for error
conditions in LSR does not help
Regards, Harald --
- Harald Welte laforge@gnumonks.org
============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)
Hi,
Harald Welte schrieb:
Just another intermediate update:
I've pushed my current debugging code to the uart-debug branch
Simply run osmocon + layer23 and the hello_world.bin image. layer23 will send a 62 byte sized HLDC message every 10 seconds to the phone, and various instances of the code will print what they see. The payload of the packet is supposed to be 0a 00 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05 06 07 00 01 02 03 04 05
but what the firmware sees is: 0a 00 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05 06 07 02 03 04 05
is this issue limited to a specific phone? I just wanted to reproduce this on my C155 but failed (l1a_l23_rx_cb returns exactly what was sent by layer23). Is this expected or am I doing something wrong?
Regards,
Florian Tobias Schandinat
Harald Welte wrote:
Hi!
It seems we're loosing serial characters on their way from the (Linux) PC to the phone.
I had problems with this during bootloader development and did some research, as did you. I came to similar conclusions (not seeing the loss in sercomm or other software but in between the two uart drivers).
One thing that I noticed while checking the datasheets is that the audio frontend for the headphone jack is never detached from the port. So I believed that this was caused by specific data patterns, which trigger a frequency response in the audio frontend. However, the fact that you have been able to reproduce this with long inter-word delays suggests that this is probably not true.
I must note though that I have not only been getting lost bytes but also reversed bytes. We spent a while discussing this in IRC. steve-m could reproduce this perfectly, as could I. The problem seems to be data-dependend though, as sending the same frame twice (with the bootloader tool) yields the same garbling.
I think I've seen this at some point in the past, but now I'm seeing it constantly.
Let's assume we're trying to send the following sequence of bytes from the PC side to the phone in one of the HLDC connections:
send_to_phone(dlci=5): 0a 00 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
what we actually get from the sercomm callback on the phone is:
l1a_l23_rx_cb: 0a 00 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
as you can see, there is a '00 01' missing between the '00' and the '02'.
Interestingly, putting a usleep(10000) after every serial write does not solve the problem, so it is not timing related. We don't get any UART overflows on the calypso, either.
If I change the data pattern, then the first bytes are transmitted fine, but as soon as there is one 00 bytes, two characters are missing from the stream.
If I strace osmocon, it is clear that we write every character, including the null-bytes, and all of the writes are successfully.
I've again looked at (and played with) the termios settings, but could not find any problem (nor solution).
It's also not the sercomm layer that is loosing bytes, as I the characters appear already lost when printing them from within the calypso UART driver, as you can see here:
getchar_nb(1) = 7e // flag character getchar_nb(1) = 05 // hdlc getchar_nb(1) = 03 // hdlc getchar_nb(1) = 0a // l1ctl_hdr (undefined msg_type 10) getchar_nb(1) = 00 // padding getchar_nb(1) = 02 // here should be 00, 01, 02... getchar_nb(1) = 03 getchar_nb(1) = 04 getchar_nb(1) = 05 [...]
Any ideas? It's really strange and I don't really know what else to do. I'm quite sure our binaries contain multiple successive null-bytes and their download is working great...
Regards, Harald
Ingo Albrecht wrote:
Harald Welte wrote:
Hi!
I must note though that I have not only been getting lost bytes but also reversed bytes. We spent a while discussing this in IRC. steve-m could reproduce this perfectly, as could I. The problem seems to be data-dependend though, as sending the same frame twice (with the bootloader tool) yields the same garbling.
To clarify: "reversed bytes" means that "004000" becomes "400000".
I think I've seen this at some point in the past, but now I'm seeing it constantly.
Let's assume we're trying to send the following sequence of bytes from the PC side to the phone in one of the HLDC connections:
send_to_phone(dlci=5): 0a 00 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
what we actually get from the sercomm callback on the phone is:
l1a_l23_rx_cb: 0a 00 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d
as you can see, there is a '00 01' missing between the '00' and the '02'.
Interestingly, putting a usleep(10000) after every serial write does not solve the problem, so it is not timing related. We don't get any UART overflows on the calypso, either.
If I change the data pattern, then the first bytes are transmitted fine, but as soon as there is one 00 bytes, two characters are missing from the stream.
If I strace osmocon, it is clear that we write every character, including the null-bytes, and all of the writes are successfully.
I've again looked at (and played with) the termios settings, but could not find any problem (nor solution).
It's also not the sercomm layer that is loosing bytes, as I the characters appear already lost when printing them from within the calypso UART driver, as you can see here:
getchar_nb(1) = 7e // flag character getchar_nb(1) = 05 // hdlc getchar_nb(1) = 03 // hdlc getchar_nb(1) = 0a // l1ctl_hdr (undefined msg_type 10) getchar_nb(1) = 00 // padding getchar_nb(1) = 02 // here should be 00, 01, 02... getchar_nb(1) = 03 getchar_nb(1) = 04 getchar_nb(1) = 05 [...]
Any ideas? It's really strange and I don't really know what else to do. I'm quite sure our binaries contain multiple successive null-bytes and their download is working great...
Regards, Harald
Hi prom,
On Sun, Mar 21, 2010 at 11:42:50PM +0100, Ingo Albrecht wrote:
Harald Welte wrote:
Hi!
It seems we're loosing serial characters on their way from the (Linux) PC to the phone.
I had problems with this during bootloader development and did some research, as did you. I came to similar conclusions (not seeing the loss in sercomm or other software but in between the two uart drivers).
One thing that I noticed while checking the datasheets is that the audio frontend for the headphone jack is never detached from the port.
There are SPDT (single-pole dual-throw) switches between the audio jack and the audio port of the calypso. Using those switches (NLASB3157DFT2), controlled by Calypso IO2, the headphone jack is exclusively used by either the UART or the audio part, so I don't understand your observation.
The only part that seems permanently connected is the HSMICBIAS from Iota. Maybe we should double-check that we don't apply a bias voltage here.
I must note though that I have not only been getting lost bytes but also reversed bytes. We spent a while discussing this in IRC. steve-m could reproduce this perfectly, as could I. The problem seems to be data-dependend though, as sending the same frame twice (with the bootloader tool) yields the same garbling.
this is even weirder. However, as you indicated in your previous mail, the trigger pattern in this case also includes a number of zero-bytes...
One of my idaes was BREAK-detection, as BREAK is nothing but a longer-than-a-byte zero level on RS232. So in theory, two successive zero bytes without the line going high in between should be recognized as break. However, the Calypso UART does not indicate any BREAK interrupt either :(
Regards, Harald
btw, i assume you considered using the HDLC escape sequence: 0x7d 0x20 as a workaround for this problem.
is '00 01' really the only sequence that is dropped? what about 00 02 or 00 80 or 00 00 00 00 ?
... i don't have a working test setup yet, got a phone, but still waiting for a serial cable. ( which should arrive this week )
willem
On Mon, Mar 22, 2010 at 01:42:36AM +0100, willem wrote:
btw, i assume you considered using the HDLC escape sequence: 0x7d 0x20 as a workaround for this problem.
no, this didn't really come to my mind so far. But of course it's a great idea as a workaround. Also, the overhead should be relatively minimal...
Will try it later today... progress is relatively low due to holidays at the moment...
Hi Willem,
On Fri, Mar 26, 2010 at 09:56:12AM +0800, Harald Welte wrote:
On Mon, Mar 22, 2010 at 01:42:36AM +0100, willem wrote:
btw, i assume you considered using the HDLC escape sequence: 0x7d 0x20 as a workaround for this problem.
no, this didn't really come to my mind so far. But of course it's a great idea as a workaround. Also, the overhead should be relatively minimal...
Will try it later today... progress is relatively low due to holidays at the moment...
It is working as expected, I've committed the workaround to the repository and am now continuing with actual code, like L23-triggered power measurements and the like.
My immediate goal is to have a small program that can be used with l1test.bin, which will iterate over all cells in all networks that it finds, and receive the BCCH+CCCH for a user-configurable time. In the resulting PCAP file we then have all SI's of all cells at the current location.
Regards, Harald
baseband-devel@lists.osmocom.org
-
Christian Vogel -
Erik Ekman -
Florian Tobias Schandinat -
Harald Welte -
Hopsing K -
Ingo Albrecht -
willem