Hi All,
On one of our sites osmo-nitb was hanging a few seconds after startup. It took me a while to track it down, and I'm sorry I do not have log output as it has gone to scroll buffer heaven, and I do not want to break the system again right now to get another output, but I can say more or less what happens.
If I see it again, I'll be sure to capture full debug log. I also have a backup of the state of the offending hlr, so I can bring it up locally and see if I can replicate and send the log. In fact, there was nothing out of the ordinary in the log anyway before nitb simply stops responding. (CTRL-C still shuts down cleanly if would seem)
This 'bad' hlr has some 30,815 subscriber entries, one of them has empty string for extension.
(We normally never get to anything near 30,000 as we purge inactive subscribers regularly, but the blank extension had broken our purge job)
On this blank extension issue, i'm not sure it coincides with the hang, but I noted this error followed by DBI traceback from here: http://git.osmocom.org/openbsc/tree/openbsc/src/libmsc/db.c#n54 Error was non unique value for column extension, which makes sense of course, as we don't give it a value. Also seems we don't recover from this, as |db_subscriber_alloc_exten() will never be called? |
I deleted some 28,000 subscriber entries that were not commissioned users, and the nitb then functions, so I'm at a loss to know where it might have been actually stuck, or if it might be related to the blank extension, or to the large amount of subscriber entries, or something else.
Maybe it's obvious to somebody. I have of course deleted the blank extension entry now from the hlr.
Thanks!
Keith.
Hi Keith,
I can confirm that OpenBSC becomes extremely slow and almost hangs after a certain amount of subscribers or a large number of SMS (30,000 subs sounds like a number it should happen). I'm not 100% certain, but I think this is the real reason and not the empty extension. If you have a copy of the HLR - you should be able to reproduce that just by starting an OpenBSC with this HLR database.
On Mon, Jul 25, 2016 at 8:19 AM, Keith keith@rhizomatica.org wrote:
Hi All,
On one of our sites osmo-nitb was hanging a few seconds after startup. It took me a while to track it down, and I'm sorry I do not have log output as it has gone to scroll buffer heaven, and I do not want to break the system again right now to get another output, but I can say more or less what happens.
If I see it again, I'll be sure to capture full debug log. I also have a backup of the state of the offending hlr, so I can bring it up locally and see if I can replicate and send the log. In fact, there was nothing out of the ordinary in the log anyway before nitb simply stops responding. (CTRL-C still shuts down cleanly if would seem)
This 'bad' hlr has some 30,815 subscriber entries, one of them has empty string for extension.
(We normally never get to anything near 30,000 as we purge inactive subscribers regularly, but the blank extension had broken our purge job)
On this blank extension issue, i'm not sure it coincides with the hang, but I noted this error followed by DBI traceback from here: http://git.osmocom.org/openbsc/tree/openbsc/src/libmsc/db.c#n54 Error was non unique value for column extension, which makes sense of course, as we don't give it a value. Also seems we don't recover from this, as db_subscriber_alloc_exten() will never be called?
I deleted some 28,000 subscriber entries that were not commissioned users, and the nitb then functions, so I'm at a loss to know where it might have been actually stuck, or if it might be related to the blank extension, or to the large amount of subscriber entries, or something else.
Maybe it's obvious to somebody. I have of course deleted the blank extension entry now from the hlr.
Thanks!
Keith.
-
Alexander Chemeris -
Keith