NITB: high cpu usage and "crossed" messages when SMS table grows

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/OpenBSC@lists.osmocom.org/.

Alexander Chemeris alexander.chemeris at gmail.com
Tue Mar 14 08:49:52 UTC 2017


Hi Neels,

On Mar 14, 2017 03:05, "Neels Hofmeyr" <nhofmeyr at sysmocom.de> wrote:

On Mon, Mar 13, 2017 at 11:01:57PM +0100, Keith wrote:
> I temporarily disabled a cron job we run at rhizomatica that purges the
> hlr SMS table of sent messages every day.
>
> After a few days I noticed slightly sluggish behaviour in the VTY, and
> sure enough, the nitb was consuming 100% cpu, not always, but presumably
> whenever it does a queue run.

Hmm, that's a very vague indicator. How performant is the hardware? For how
long does this load endure? Does the process hang otherwise, is service
disrupted?

>From how I got to know the SMS code, it appears to have sound safeguards in
place, e.g.  limits the number of SMS to be delivered per queue run, and
only
attempts deliveries of SMS for actually attached subscribers ... But in
fact we
don't have load testing in place. It would be good to find out where
unproportional CPU load is coming from -- SQlite? The NITB sms code? From a
theoretical standpoint I'd also expect the SMS database to discard messages
that are past a certain age, not sure though, as I'm not that deeply
familiar
with that (yet).

Would be good to know: how many SMS are pending, for how many subscribers,
of
which how many are currently attached? How often are SMS deliveries being
retried and end in failure? ... and anything else you can think of.


We looked into this a couple years ago, but didn't come up with a good
solution context switched to something else.

Just add a few thousand SMS to the DB (DB should 10Mb or more roughly) and
start OsmoNITB. You'll notice it's agony immediately.

It's been a while, but IIRC the issue is that the DB didn't have proper
indexes for the kinds of queries we're running on it, so it's getting super
inefficient.

Regarding removing SMS - it should be fine based on validity time of the
SMS and it was completely broken. I had a patch set which fixed validity
time handling, but IIRC it wasn't merged. We can probably dug it up, but I
don't have much time to rebase / adapt it to the new codebase right now. If
there are any volunteers, that would be great.

Please excuse typos. Written with a touchscreen keyboard.

--
Regards,
Alexander Chemeris
CTO/Founder Fairwaves, Inc.
https://fairwaves.co
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/openbsc/attachments/20170314/d9516a03/attachment.htm>


More information about the OpenBSC mailing list