NITB: high cpu usage and "crossed" messages when SMS table grows

[Keith]

On 14/03/2017 01:04, Neels Hofmeyr wrote:
> On Mon, Mar 13, 2017 at 11:01:57PM +0100, Keith wrote:
>> the nitb was consuming 100% cpu, not always, but presumably
>> whenever it does a queue run.
> Hmm, that's a very vague indicator. 
I know, I know.. :-/
> Would be good to know: how many SMS are pending, for how many subscribers, of
> which how many are currently attached? How often are SMS deliveries being
> retried and end in failure? ... and anything else you can think of.
Yesterday I did
 DELETE FROM SMS WHERE sent IS NOT NULL
and the cpu usage problem goes away.

Previous to that, On the site where I most noticed it:

Total entries in table: 114231
Pending (not sent): 1662
Distinct number of subscribers with pending SMS: 501

So as Alexander says, It seems to have to do with simply the amount of
entries in the table.
Alex, if you dig out those patches, I'm happy to submit them for Code
Review.
>> messages being delivered
>> to the wrong destination.
> Whoa! That should absolutely not happen. I can't see how this is even possible.
That's exactly the reaction that happened when I mentioned it at
OsmoDevCon last year. :-)
I have also been somewhat incredulous of these reports, putting it down
to user error, or possibly something in our SMPP->kannel->python
stuff->kannel->SMPP->Osmo chain, and after working a little on that
code, the problem ceased to be reported, but it's quite telling that at
the same time that I stop purging the SMS table and we grow above
100,000 entries in SMS table, we get reports from at least 4 communities
of these "crossed" messages.

k/