Change in osmo-bsc[master]: osmo_bsc_main: integrate MGW pooling into osmo-bsc

dexter gerrit-no-reply at lists.osmocom.org
Tue Aug 3 14:48:51 UTC 2021


dexter has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-bsc/+/25123 )


Change subject: osmo_bsc_main: integrate MGW pooling into osmo-bsc
......................................................................

osmo_bsc_main: integrate MGW pooling into osmo-bsc

Since the libosmo-mgcp-client now supports MGW pooling, lets use this
feature in osmo-bsc. Large RAN installations may benefit from
distributing the RTP voice stream load on multiple media gateways.

Depends: osmo-mgw Icaaba0e470e916eefddfee750b83f5f65291a6b0
Change-Id: I8f33ab2cea04b545c403a6fe479aa963a0fc0d0d
Related: SYS#5091
---
A doc/manuals/chapters/mgwpool.adoc
M doc/manuals/chapters/running.adoc
M doc/manuals/osmobsc-usermanual.adoc
M include/osmocom/bsc/gsm_data.h
M include/osmocom/bsc/vty.h
M src/osmo-bsc/bsc_subscr_conn_fsm.c
M src/osmo-bsc/bsc_vty.c
M src/osmo-bsc/osmo_bsc_main.c
8 files changed, 326 insertions(+), 17 deletions(-)



  git pull ssh://gerrit.osmocom.org:29418/osmo-bsc refs/changes/23/25123/1

diff --git a/doc/manuals/chapters/mgwpool.adoc b/doc/manuals/chapters/mgwpool.adoc
new file mode 100644
index 0000000..d7f1b1c
--- /dev/null
+++ b/doc/manuals/chapters/mgwpool.adoc
@@ -0,0 +1,240 @@
+[[mgw_pooling]]
+== MGW Pooling
+
+OsmoBSC is able to use a pool of media gateway (MGW) instances since mid 2021.
+The aim of MGW pooling is to evenly distribute the RTP voice stream load across
+multiple MGW instances. This can help to get a better CPU utilization on
+multi-core CPUs.
+
+The load distribution is managed in such a way that when a new call is placed
+the pool will automatically assign the call to the MGW with the lowest load.
+
+MGW pooling is recommended for larger RAN installations. For small networks and
+lab installations the classic method with one MGW per BSC offers sufficient
+performance.
+
+=== VTY configuration
+
+In OsmoBSC the MGW is controlled via an MGCP-Client. The VTY commands to
+configure the MGCP-Client are part of the 'msc' node due to historical reasons.
+Unfortunately this concept does not allow to configure multiple MGCP-Client
+instances as required by MGW pooling. In order to support MGW pooling a new
+'mgw' node has been added under the 'network' node.
+
+=== Legacy configuration files
+
+legacy OsmoBSC configuration files will continue to work but as soon as an MGW
+pool is configured the 'mgw' settings under the 'msc' node will be ignored.
+
+Example configuration with only one MGCP-Client under the 'msc' node:
+----
+msc 0
+ mgw remote-ip 127.0.0.1
+ mgw remote-port 2428
+----
+
+=== MGW pool configuration
+
+to setup an MGW pool, the user must first install multiple OsmoMGW instances so
+that they won’t interfere each other. This can be done using different local
+host ip-addresses or different ports. When OsmoMGW is installed from packages,
+the systemd configuration may also require adjustment.
+
+The VTY settings under the 'mgw' node work the same way as the VTY settings for
+the MGW under the 'msc' node.
+
+Example configuration with two MGCP-Client instances in a pool:
+----
+ mgw 0
+  mgw remote-ip 127.0.0.1
+  mgw remote-port 2432
+  mgw local-ip 127.0.0.1
+  mgw local-port 2431
+  mgw endpoint-domain mgw0  
+ mgw 1
+  mgw remote-ip 127.0.0.1
+  mgw remote-port 2430
+  mgw local-ip 127.0.0.1
+  mgw local-port 2429
+  mgw endpoint-domain mgw1
+----
+
+NOTE: When working with multiple MGW / MGCP-Client instances it is higly
+recommended to assign each MGW an individual domain name. Otherwise it can be
+difficult to distinguish the endpoint names in the log.
+
+=== MGW pool management
+
+While it was not possible to change the MGCP-Client configuration under the
+“msc” node at runtime, the pool is fully runtime-manageable. The only limitation
+is that an MGCP-Client can not be restarted or removed as long as it is serving
+calls.
+
+==== MGW pool status
+
+The VTY implenets a 'show mgw-pool' command that lists the currently configured
+MGW pool members, their status and call utilization.
+
+----
+OsmoBSC> show mgw-pool 
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 1
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 0
+----
+
+==== Adding an MGW / MGCP-Client to the MGW pool
+
+To add a new MGCP-Client to the pool, the 'mgw' node is used. Like with the
+'bts' or the 'msc' node a reference number is used that usually starts at 0.
+However it is still possible to assign any number from 0-255. The enumeration
+also may contain gaps.
+
+----
+OsmoBSC> enable 
+OsmoBSC# configure terminal 
+OsmoBSC(config)# network 
+OsmoBSC(config-net)# mgw 2
+OsmoBSC(config-mgw)# mgw
+  local-ip         local bind to connect to MGW from
+  local-port       local port to connect to MGW from
+  remote-ip        remote IP address to reach the MGW at
+  remote-port      remote port to reach the MGW at
+  endpoint-domain  Set the domain name to send in MGCP messages, e.g. the part 'foo' in 'rtpbridge/*@foo'.
+  reset-endpoint   Add an endpoint name that should be reset (DLCX) on connect to the reset-endpoint list,e.g. 'rtpbridge/*'
+----
+
+The newly added MGW will immediately appear in the mgc-pool list but it won't
+be used until its configuration finished by restarting it.
+
+----
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 2
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 3
+%  MGW 2
+%   mgcp-client:   disconnected
+%   service:       unblocked
+%   ongoing calls: 0
+----
+
+==== Reconnecting an MGW / MGCP-Client
+
+It may become necessary to reconnect an MGCP-Client. This is the case when the
+VTY configuration was changed at runtime. In order to make the changes effective
+the MGW configuration must be reloaded by reconnecting the MGW connection. Also
+newly created MGW instances require a reconnect once their configuration is
+done.
+
+To reconnect an MGCP-Client use the 'reconnect' VTY command:
+----
+OsmoBSC# mgw 2 reconnect 
+----
+
+The mgcp-client status should immediately change to 'connected'. The MGW is now
+ready to be used for new calls.
+
+----
+OsmoBSC# show mgw-pool 
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 2
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 3
+%  MGW 2
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 0
+----
+
+It should be noted that MGCP a protocol is used via UDP, the connect only
+happens locally to forward the UDP datagrams properly. Also (unless a reset
+endpoint is configured like in the example config above) there will be no
+immediate interaction with the MGW. However, the log should at least confirm
+the the connect worked and the MGCP client has been created successfully.
+
+----
+Mon Aug  2 17:15:00 2021 DLMGCP <0025> mgcp_client.c:788 MGCP client: using endpoint domain '@mgw'
+Mon Aug  2 17:15:00 2021 DLMGCP <0025> mgcp_client.c:908 MGCP GW connection: r=127.0.0.1:2427<->l=127.0.0.1:2727
+----
+
+It is strongly advised to check the activity on the related MGW and to follow
+the log in order to see that the communication between OsmoBSC and the MGW is
+working correctly.
+
+==== Blocking an MGW / MGCP-Client
+
+If it becomes apparent that an MGCP-Client must be restarted or removed from
+the config (maintenance) the operator can put that MGCP-Client into a blocked
+mode. A blocked MGCP-Client will still serve the ongoing calls but it will not
+be picked for the assignment of new calls.
+
+To block an MGCP-Client use the 'block' VTY command:
+----
+OsmoBSC# mgw 2 block 
+OsmoBSC# show mgw-pool 
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 11
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 12
+%  MGW 2
+%   mgcp-client:   connected
+%   service:       blocked
+%   ongoing calls: 10
+----
+
+When the number of ongoing calls has tapered off, the MGW / MGCP-Client can be
+restarted or removed if necessary.
+
+----
+OsmoBSC# show mgw-pool 
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 15
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 14
+%  MGW 2
+%   mgcp-client:   connected
+%   service:       blocked
+%   ongoing calls: 0
+----
+
+If the blockade should be reverted, the 'unblock' VTY command can be used in
+the same way to remove the blockade. (Reconnecting will not remove the
+blockade.)
+
+==== Removing an MGW / MGCP-Client
+
+An MGCP-Client is removed from the pool using the 'no mgw' command from the
+configure terminal. The MGCP-Client instance will automatically be terminated
+and the related resources are freed. The only requirement is that there are no
+ongoing calls on the selected instance. 
+
+----
+OsmoBSC# configure terminal 
+OsmoBSC(config)# network 
+OsmoBSC(config-net)# no mgw 2
+----
diff --git a/doc/manuals/chapters/running.adoc b/doc/manuals/chapters/running.adoc
index 327e0bd..26d62f6 100644
--- a/doc/manuals/chapters/running.adoc
+++ b/doc/manuals/chapters/running.adoc
@@ -143,6 +143,9 @@
 DLCX to the media gateway. This helps to clear lingering calls from the
 media gateway when the OsmoBSC is restarted.
 
+NOTE: OsmoBSC is also able to handle a pool of media gateways for load
+distribution. See also See also <<mgw_pooling>>.
+
 
 ==== Configure Lb to connect to an SMLC
 
diff --git a/doc/manuals/osmobsc-usermanual.adoc b/doc/manuals/osmobsc-usermanual.adoc
index c7589cc..120fe2c 100644
--- a/doc/manuals/osmobsc-usermanual.adoc
+++ b/doc/manuals/osmobsc-usermanual.adoc
@@ -32,6 +32,8 @@
 
 include::{srcdir}/chapters/mscpool.adoc[]
 
+include::{srcdir}/chapters/mgwpool.adoc[]
+
 include::{srcdir}/chapters/smlc.adoc[]
 
 include::./common/chapters/qos-dscp-pcp.adoc[]
diff --git a/include/osmocom/bsc/gsm_data.h b/include/osmocom/bsc/gsm_data.h
index 5a20e72..8fd88cb 100644
--- a/include/osmocom/bsc/gsm_data.h
+++ b/include/osmocom/bsc/gsm_data.h
@@ -1346,8 +1346,14 @@
 	struct osmo_timer_list bts_store_uptime_timer;
 
 	struct {
+		/* Legacy MGCP client configuration under msc node (also required for
+		 * MGCP proxy when sccp-lite is used) */
 		struct mgcp_client_conf *conf;
-		struct mgcp_client *client;
+
+		/* MGW pool, also contains the legacy MGCP client. */
+		struct mgcp_client_pool *mgw_pool;
+
+		/* Timer definitions, the same for all MGW pool members */
 		struct osmo_tdef *tdefs;
 	} mgw;
 
diff --git a/include/osmocom/bsc/vty.h b/include/osmocom/bsc/vty.h
index 9ea9835..70b973d 100644
--- a/include/osmocom/bsc/vty.h
+++ b/include/osmocom/bsc/vty.h
@@ -23,6 +23,7 @@
 
 enum bsc_vty_node {
 	GSMNET_NODE = _LAST_OSMOVTY_NODE + 1,
+        MGW_NODE,
 	BTS_NODE,
 	TRX_NODE,
 	TS_NODE,
diff --git a/src/osmo-bsc/bsc_subscr_conn_fsm.c b/src/osmo-bsc/bsc_subscr_conn_fsm.c
index afb633d..2256487 100644
--- a/src/osmo-bsc/bsc_subscr_conn_fsm.c
+++ b/src/osmo-bsc/bsc_subscr_conn_fsm.c
@@ -45,6 +45,7 @@
 #include <osmocom/bsc/assignment_fsm.h>
 #include <osmocom/bsc/codec_pref.h>
 #include <osmocom/mgcp_client/mgcp_client_endpoint_fsm.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 #include <osmocom/core/byteswap.h>
 #include <osmocom/bsc/lb.h>
 #include <osmocom/bsc/lcs_loc_req.h>
@@ -518,39 +519,54 @@
 						uint16_t msc_assigned_cic, struct gsm_lchan *for_lchan)
 {
 	const char *epname;
+	struct mgcp_client *mgcp_client = NULL;
 
 	if (conn->user_plane.mgw_endpoint)
 		return conn->user_plane.mgw_endpoint;
 
 	if (gscon_is_sccplite(conn)) {
+		/* Get MGCP client from pool */
+		mgcp_client = mgcp_client_pool_get(conn->network->mgw.mgw_pool);
+		if (!mgcp_client) {
+			LOGPFSML(conn->fi, LOGL_ERROR,
+				 "cannot ensure MGW endpoint -- no MGW configured, check configuration!\n");
+			conn->user_plane.mgw_endpoint = NULL;
+			return NULL;
+		}
+
 		/* derive endpoint name from CIC on A interface side */
 		conn->user_plane.mgw_endpoint =
 			osmo_mgcpc_ep_alloc(conn->fi, GSCON_EV_FORGET_MGW_ENDPOINT,
-					    conn->network->mgw.client,
+					    mgcp_client,
 					    conn->network->mgw.tdefs,
 					    conn->fi->id,
 					    "%x@%s", msc_assigned_cic,
-					    mgcp_client_endpoint_domain(conn->network->mgw.client));
+					    mgcp_client_endpoint_domain(mgcp_client));
 		LOGPFSML(conn->fi, LOGL_DEBUG, "MGW endpoint name derived from CIC 0x%x: %s\n",
 			 msc_assigned_cic, osmo_mgcpc_ep_name(conn->user_plane.mgw_endpoint));
 
 	} else if (gscon_is_aoip(conn)) {
+		/* Get MGCP client from pool */
+		mgcp_client = mgcp_client_pool_get(conn->network->mgw.mgw_pool);
+		if (!mgcp_client) {
+			LOGPFSML(conn->fi, LOGL_ERROR,
+				 "cannot ensure MGW endpoint -- no MGW configured, check configuration!\n");
+			conn->user_plane.mgw_endpoint = NULL;
+			return NULL;
+		}
 
 		if (is_ipaccess_bts(for_lchan->ts->trx->bts))
 			/* use dynamic RTPBRIDGE endpoint allocation in MGW */
-			epname = mgcp_client_rtpbridge_wildcard(conn->network->mgw.client);
+			epname = mgcp_client_rtpbridge_wildcard(mgcp_client);
 		else {
-			epname = mgcp_client_e1_epname(conn, conn->network->mgw.client, for_lchan->ts->e1_link.e1_nr,
+			epname = mgcp_client_e1_epname(conn, mgcp_client, for_lchan->ts->e1_link.e1_nr,
 						       for_lchan->ts->e1_link.e1_ts, 16,
 						       for_lchan->ts->e1_link.e1_ts_ss*2);
 		}
 
 		conn->user_plane.mgw_endpoint =
-			osmo_mgcpc_ep_alloc(conn->fi, GSCON_EV_FORGET_MGW_ENDPOINT,
-					    conn->network->mgw.client,
-					    conn->network->mgw.tdefs,
-					    conn->fi->id,
-					    "%s", epname);
+			osmo_mgcpc_ep_alloc(conn->fi, GSCON_EV_FORGET_MGW_ENDPOINT, mgcp_client,
+					    conn->network->mgw.tdefs, conn->fi->id, "%s", epname);
 	} else {
 		LOGPFSML(conn->fi, LOGL_ERROR, "Conn is neither SCCPlite nor AoIP!?\n");
 		return NULL;
@@ -801,6 +817,12 @@
 
 static void gscon_forget_mgw_endpoint(struct gsm_subscriber_connection *conn)
 {
+	struct mgcp_client *mgcp_client;
+
+	/* Put MGCP client back into MGW pool */
+	mgcp_client = osmo_mgcpc_ep_client(conn->user_plane.mgw_endpoint);
+	mgcp_client_pool_put(mgcp_client);
+
 	conn->user_plane.mgw_endpoint = NULL;
 	conn->user_plane.mgw_endpoint_ci_msc = NULL;
 	conn->ho.created_ci_for_msc = NULL;
diff --git a/src/osmo-bsc/bsc_vty.c b/src/osmo-bsc/bsc_vty.c
index 6d10426..d3d769e 100644
--- a/src/osmo-bsc/bsc_vty.c
+++ b/src/osmo-bsc/bsc_vty.c
@@ -37,6 +37,7 @@
 #include <osmocom/gsm/gsm0502.h>
 
 #include <osmocom/mgcp_client/mgcp_client_endpoint_fsm.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include <osmocom/bsc/vty.h>
 #include <osmocom/bsc/gsm_data.h>
@@ -3382,7 +3383,9 @@
 	install_element(GSMNET_NODE, &cfg_net_nri_null_add_cmd);
 	install_element(GSMNET_NODE, &cfg_net_nri_null_del_cmd);
 
+
 	bts_vty_init();
+	mgcp_client_vty_init_pool(GSMNET_NODE, MGW_NODE, " ", vty_global_gsm_network->mgw.mgw_pool);
 
 	install_element(ENABLE_NODE, &drop_bts_cmd);
 	install_element(ENABLE_NODE, &restart_bts_cmd);
diff --git a/src/osmo-bsc/osmo_bsc_main.c b/src/osmo-bsc/osmo_bsc_main.c
index b97c7a1..d44d3b7 100644
--- a/src/osmo-bsc/osmo_bsc_main.c
+++ b/src/osmo-bsc/osmo_bsc_main.c
@@ -57,6 +57,7 @@
 #include <osmocom/vty/cpu_sched_vty.h>
 
 #include <osmocom/mgcp_client/mgcp_client_endpoint_fsm.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include <osmocom/abis/abis.h>
 #include <osmocom/bsc/abis_om2000.h>
@@ -69,6 +70,7 @@
 #include <osmocom/bsc/bts.h>
 
 #include <osmocom/mgcp_client/mgcp_client.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include <osmocom/sigtran/xua_msg.h>
 
@@ -866,6 +868,41 @@
 extern void *tall_tqe_ctx;
 extern void *tall_ctr_ctx;
 
+static int bsc_mgw_setup()
+{
+	struct mgcp_client *mgcp_client_legacy;
+	unsigned int pool_members_initalized;
+
+	/* Initalize MGW pool. This initalizes and connects all MGCP clients that are currently configured in
+	 * the pool. Adding additional MGCP clients to the pool is possible but the user has to initalize and
+	 * start them manually from the VTY. */
+	pool_members_initalized = mgcp_client_connect_pool(bsc_gsmnet->mgw.mgw_pool);
+	if (pool_members_initalized) {
+		LOGP(DNM, LOGL_NOTICE,
+		     "MGW pool with %u pool members configured, (ignoring MGW configuration in VTY node 'msc').\n",
+		     pool_members_initalized);
+		return 0;
+	}
+
+	/* Initalize and connect a legacy MGCP client. This MGCP client will appear as the one and only pool
+	 * member if there is no MGW pool configured. */
+	LOGP(DNM, LOGL_NOTICE, "No MGW pool configured, using MGW configuration in VTY node 'msc'\n");
+        mgcp_client_legacy = mgcp_client_init(bsc_gsmnet, bsc_gsmnet->mgw.conf);
+	if (!mgcp_client_legacy) {
+		LOGP(DNM, LOGL_ERROR, "MGW (legacy) client initalization failed\n");
+		return -EINVAL;
+	}
+	if (mgcp_client_connect(mgcp_client_legacy)) {
+		LOGP(DNM, LOGL_ERROR, "MGW (legacy) connect failed at (%s:%u)\n",
+		     bsc_gsmnet->mgw.conf->remote_addr,
+		     bsc_gsmnet->mgw.conf->remote_port);
+		return -EINVAL;
+	}
+	mgcp_client_pool_register_legacy(bsc_gsmnet->mgw.mgw_pool, mgcp_client_legacy);
+
+	return 0;
+}
+
 int main(int argc, char **argv)
 {
 	struct bsc_msc_data *msc;
@@ -896,6 +933,7 @@
 	}
 
 	bsc_gsmnet->mgw.conf = talloc_zero(bsc_gsmnet, struct mgcp_client_conf);
+	bsc_gsmnet->mgw.mgw_pool = mgcp_client_alloc_pool(bsc_gsmnet);
 	mgcp_client_conf_init(bsc_gsmnet->mgw.conf);
 
 	bts_init();
@@ -999,14 +1037,8 @@
 		}
 	}
 
-	bsc_gsmnet->mgw.client = mgcp_client_init(bsc_gsmnet, bsc_gsmnet->mgw.conf);
-
-	if (mgcp_client_connect(bsc_gsmnet->mgw.client)) {
-		LOGP(DNM, LOGL_ERROR, "MGW connect failed at (%s:%u)\n",
-		     bsc_gsmnet->mgw.conf->remote_addr,
-		     bsc_gsmnet->mgw.conf->remote_port);
+	if (bsc_mgw_setup() != 0)
 		exit(1);
-	}
 
 	if (osmo_bsc_sigtran_init(&bsc_gsmnet->mscs) != 0) {
 		LOGP(DNM, LOGL_ERROR, "Failed to initialize sigtran backhaul.\n");

-- 
To view, visit https://gerrit.osmocom.org/c/osmo-bsc/+/25123
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings

Gerrit-Project: osmo-bsc
Gerrit-Branch: master
Gerrit-Change-Id: I8f33ab2cea04b545c403a6fe479aa963a0fc0d0d
Gerrit-Change-Number: 25123
Gerrit-PatchSet: 1
Gerrit-Owner: dexter <pmaier at sysmocom.de>
Gerrit-MessageType: newchange
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/gerrit-log/attachments/20210803/33116fbc/attachment.htm>


More information about the gerrit-log mailing list