<p>Vadim Yanitskiy <strong>merged</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/osmo-hlr/+/14939">View Change</a></p><div style="white-space:pre-wrap">Approvals:
laforge: Looks good to me, approved
neels: Looks good to me, but someone else must approve
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">src/db.c: integrate SQLite3 with talloc allocator<br><br>This change introduces an optional feature that allows to make<br>SQLite3 use talloc for all internal allocations. This would<br>facilitate finding memleaks. OsmoHLR needs to be configured<br>with '--enable-sqlite-talloc'.<br><br> full talloc report on 'OsmoHLR' (total 292168 bytes in 449 blocks)<br> struct osmo_gsup_server contains 162 bytes in 3 blocks (ref 0)<br> ...<br> struct db_context contains 288407 bytes in 420 blocks (ref 0)<br> hlr.db contains 7 bytes in 1 blocks (ref 0)<br> SQLite3 contains 288192 bytes in 418 blocks (ref 0)<br> db.c:95 contains 48 bytes in 1 blocks (ref 0)<br> db.c:95 contains 2 bytes in 1 blocks (ref 0)<br> ...<br><br>Unfortunately, old SQLite3 versions (such as 3.8.2) run out<br>of memory when trying to initialize a new database:<br><br> DDB ERROR db.c:88 (7) statement aborts at 3: []<br> DDB ERROR db.c:420 Unable to set Write-Ahead Logging: out of memory<br> DDB ERROR db.c:88 (7) statement aborts at 3: []<br> DDB ERROR db.c:238 Unable to prepare SQL statement<br> 'SELECT name FROM sqlite_master WHERE type='table' AND name=?'<br> ...<br><br>I've noticed a huge difference in heap usage footprint compared to<br>generic malloc. At the same time, the recent versions (at least<br>3.24.0), work just fine.<br><br>Change-Id: Icfe67ed0f063b63e6794f9516da3003d01cf20a7<br>---<br>M configure.ac<br>M src/Makefile.am<br>M src/db.c<br>M src/db.h<br>A src/db_debug.c<br>M tests/db/Makefile.am<br>6 files changed, 126 insertions(+), 0 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/configure.ac b/configure.ac</span><br><span>index 6694f80..ef703f2 100644</span><br><span>--- a/configure.ac</span><br><span>+++ b/configure.ac</span><br><span>@@ -59,6 +59,21 @@</span><br><span> CPPFLAGS="$CPPFLAGS -fsanitize=address -fsanitize=undefined"</span><br><span> fi</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+AC_ARG_ENABLE([sqlite_talloc],</span><br><span style="color: hsl(120, 100%, 40%);">+ AC_HELP_STRING([--enable-sqlite-talloc],</span><br><span style="color: hsl(120, 100%, 40%);">+ [Configure SQLite3 to use talloc memory allocator [default=no]]),</span><br><span style="color: hsl(120, 100%, 40%);">+ [sqlite_talloc="$enableval"],[sqlite_talloc="no"])</span><br><span style="color: hsl(120, 100%, 40%);">+if test "x$sqlite_talloc" = "xyes" ; then</span><br><span style="color: hsl(120, 100%, 40%);">+ # Older versions of SQLite3 (at least 3.8.2) become unstable with talloc.</span><br><span style="color: hsl(120, 100%, 40%);">+ # Feel free to relax to 3.24.0 > VER > 3.8.2 if it works for you.</span><br><span style="color: hsl(120, 100%, 40%);">+ # FIXME: PKG_CHECK_MODULES() may return cached result here!</span><br><span style="color: hsl(120, 100%, 40%);">+ PKG_CHECK_MODULES(SQLITE3, sqlite3 >= 3.24.0)</span><br><span style="color: hsl(120, 100%, 40%);">+ AC_DEFINE([SQLITE_USE_TALLOC], 1, [Use talloc for SQLite3])</span><br><span style="color: hsl(120, 100%, 40%);">+fi</span><br><span style="color: hsl(120, 100%, 40%);">+AC_MSG_CHECKING([whether to use talloc for SQLite3])</span><br><span style="color: hsl(120, 100%, 40%);">+AC_MSG_RESULT([$sqlite_talloc])</span><br><span style="color: hsl(120, 100%, 40%);">+AM_CONDITIONAL([DB_SQLITE_DEBUG], [test "x$sqlite_talloc" = "xyes"])</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> AC_ARG_ENABLE(werror,</span><br><span> [AS_HELP_STRING(</span><br><span> [--enable-werror],</span><br><span>diff --git a/src/Makefile.am b/src/Makefile.am</span><br><span>index 131b44f..a042e4e 100644</span><br><span>--- a/src/Makefile.am</span><br><span>+++ b/src/Makefile.am</span><br><span>@@ -97,6 +97,11 @@</span><br><span> $(LIBOSMOGSM_LIBS) \</span><br><span> $(NULL)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+if DB_SQLITE_DEBUG</span><br><span style="color: hsl(120, 100%, 40%);">+osmo_hlr_SOURCES += db_debug.c</span><br><span style="color: hsl(120, 100%, 40%);">+osmo_hlr_db_tool_SOURCES += db_debug.c</span><br><span style="color: hsl(120, 100%, 40%);">+endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> BOOTSTRAP_SQL = $(top_srcdir)/sql/hlr.sql</span><br><span> </span><br><span> db_bootstrap.h: $(BOOTSTRAP_SQL) $(srcdir)/db_sql2c.sed</span><br><span>diff --git a/src/db.c b/src/db.c</span><br><span>index 7de61a2..5e6b5eb 100644</span><br><span>--- a/src/db.c</span><br><span>+++ b/src/db.c</span><br><span>@@ -365,6 +365,17 @@</span><br><span> LOGP(DDB, LOGL_INFO, "Compiled against SQLite3 lib version %s\n", SQLITE_VERSION);</span><br><span> LOGP(DDB, LOGL_INFO, "Running with SQLite3 lib version %s\n", sqlite3_libversion());</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+#ifdef SQLITE_USE_TALLOC</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Configure SQLite3 to use talloc memory allocator */</span><br><span style="color: hsl(120, 100%, 40%);">+ rc = db_sqlite3_use_talloc(ctx);</span><br><span style="color: hsl(120, 100%, 40%);">+ if (rc == SQLITE_OK) {</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGP(DDB, LOGL_NOTICE, "SQLite3 is configured to use talloc\n");</span><br><span style="color: hsl(120, 100%, 40%);">+ } else {</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGP(DDB, LOGL_ERROR, "Failed to configure SQLite3 "</span><br><span style="color: hsl(120, 100%, 40%);">+ "to use talloc, using default memory allocator\n");</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> dbc->fname = talloc_strdup(dbc, fname);</span><br><span> </span><br><span> for (i = 0; i < 0xfffff; i++) {</span><br><span>diff --git a/src/db.h b/src/db.h</span><br><span>index 6e4bf49..15d83de 100644</span><br><span>--- a/src/db.h</span><br><span>+++ b/src/db.h</span><br><span>@@ -39,6 +39,11 @@</span><br><span> sqlite3_stmt *stmt[_NUM_DB_STMT];</span><br><span> };</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+/* Optional feature to make SQLite3 using talloc */</span><br><span style="color: hsl(120, 100%, 40%);">+#ifdef SQLITE_USE_TALLOC</span><br><span style="color: hsl(120, 100%, 40%);">+int db_sqlite3_use_talloc(void *ctx);</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> void db_remove_reset(sqlite3_stmt *stmt);</span><br><span> bool db_bind_text(sqlite3_stmt *stmt, const char *param_name, const char *text);</span><br><span> bool db_bind_int(sqlite3_stmt *stmt, const char *param_name, int nr);</span><br><span>diff --git a/src/db_debug.c b/src/db_debug.c</span><br><span>new file mode 100644</span><br><span>index 0000000..13ccdd6</span><br><span>--- /dev/null</span><br><span>+++ b/src/db_debug.c</span><br><span>@@ -0,0 +1,86 @@</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * libtalloc based memory allocator for SQLite3.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * (C) 2019 by Vadim Yanitskiy <axilirator@gmail.com></span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * All Rights Reserved</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is free software; you can redistribute it and/or modify</span><br><span style="color: hsl(120, 100%, 40%);">+ * it under the terms of the GNU Affero General Public License as published by</span><br><span style="color: hsl(120, 100%, 40%);">+ * the Free Software Foundation; either version 3 of the License, or</span><br><span style="color: hsl(120, 100%, 40%);">+ * (at your option) any later version.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is distributed in the hope that it will be useful,</span><br><span style="color: hsl(120, 100%, 40%);">+ * but WITHOUT ANY WARRANTY; without even the implied warranty of</span><br><span style="color: hsl(120, 100%, 40%);">+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span><br><span style="color: hsl(120, 100%, 40%);">+ * GNU Affero General Public License for more details.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * You should have received a copy of the GNU Affero General Public License</span><br><span style="color: hsl(120, 100%, 40%);">+ * along with this program. If not, see <http://www.gnu.org/licenses/>.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#include <sqlite3.h></span><br><span style="color: hsl(120, 100%, 40%);">+#include <talloc.h></span><br><span style="color: hsl(120, 100%, 40%);">+#include <errno.h></span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* Dedicated talloc context for SQLite */</span><br><span style="color: hsl(120, 100%, 40%);">+static void *db_sqlite_ctx = NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+static void *tall_xMalloc(int size)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ return talloc_size(db_sqlite_ctx, size);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+static void tall_xFree(void *ptr)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ talloc_free(ptr);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+static void *tall_xRealloc(void *ptr, int size)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ return talloc_realloc_fn(db_sqlite_ctx, ptr, size);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+static int tall_xSize(void *ptr)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ return talloc_total_size(ptr);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* DUMMY: talloc doesn't round up the allocation size */</span><br><span style="color: hsl(120, 100%, 40%);">+static int tall_xRoundup(int size) { return size; }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* DUMMY: nothing to initialize */</span><br><span style="color: hsl(120, 100%, 40%);">+static int tall_xInit(void *data) { return 0; }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* DUMMY: nothing to deinitialize */</span><br><span style="color: hsl(120, 100%, 40%);">+static void tall_xShutdown(void *data) { }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* Interface between SQLite and talloc memory allocator */</span><br><span style="color: hsl(120, 100%, 40%);">+static const struct sqlite3_mem_methods tall_sqlite_if = {</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Memory allocation function */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xMalloc = &tall_xMalloc,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Free a prior allocation */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xFree = &tall_xFree,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Resize an allocation */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xRealloc = &tall_xRealloc,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Return the size of an allocation */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xSize = &tall_xSize,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Round up request size to allocation size */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xRoundup = &tall_xRoundup,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Initialize the memory allocator */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xInit = &tall_xInit,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Deinitialize the memory allocator */</span><br><span style="color: hsl(120, 100%, 40%);">+ .xShutdown = &tall_xShutdown,</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Argument to xInit() and xShutdown() */</span><br><span style="color: hsl(120, 100%, 40%);">+ .pAppData = NULL,</span><br><span style="color: hsl(120, 100%, 40%);">+};</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+int db_sqlite3_use_talloc(void *ctx)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ if (db_sqlite_ctx != NULL)</span><br><span style="color: hsl(120, 100%, 40%);">+ return -EEXIST;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ db_sqlite_ctx = talloc_named_const(ctx, 0, "SQLite3");</span><br><span style="color: hsl(120, 100%, 40%);">+ return sqlite3_config(SQLITE_CONFIG_MALLOC, &tall_sqlite_if);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span>diff --git a/tests/db/Makefile.am b/tests/db/Makefile.am</span><br><span>index fa925f8..5730937 100644</span><br><span>--- a/tests/db/Makefile.am</span><br><span>+++ b/tests/db/Makefile.am</span><br><span>@@ -36,6 +36,10 @@</span><br><span> $(SQLITE3_LIBS) \</span><br><span> $(NULL)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+if DB_SQLITE_DEBUG</span><br><span style="color: hsl(120, 100%, 40%);">+db_test_LDADD += $(top_builddir)/src/db_debug.o</span><br><span style="color: hsl(120, 100%, 40%);">+endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> .PHONY: db_test.db update_exp manual manual-nonverbose manual-gdb</span><br><span> db_test.db:</span><br><span> rm -f db_test.db</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-hlr/+/14939">change 14939</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-hlr/+/14939"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-hlr </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: Icfe67ed0f063b63e6794f9516da3003d01cf20a7 </div>
<div style="display:none"> Gerrit-Change-Number: 14939 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: Vadim Yanitskiy <axilirator@gmail.com> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: Vadim Yanitskiy <axilirator@gmail.com> </div>
<div style="display:none"> Gerrit-Reviewer: laforge <laforge@gnumonks.org> </div>
<div style="display:none"> Gerrit-Reviewer: neels <nhofmeyr@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>