<p>laforge <strong>submitted</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/libosmocore/+/19372">View Change</a></p><div style="white-space:pre-wrap">Approvals:
laforge: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">libomsocoding: NEON viterbi acceleration<br><br>configure flag required to enable this: --enable-neon<br><br>Although autodetection according to __ARM_NEON would work because this<br>is only defined if the fpu is neon neon-fp16 neon-vfpv3 neon-vfpv4<br>neon-fp-armv8 crypto-neon-fp-armv8 doing that would lead to a unknown<br>performance impact, so it needs to be enabled manually.<br><br>Speedup is about ~1.3-1.5 on a unspecified single core Cortex A9. This<br>requires handling a special case for RACH with len 14 which is far too<br>short for neon and would actually incur a performance penalty of 25%.<br><br>Related: OS#4585<br>Change-Id: I58ff2cb4ce3514f43390ff0a2121f81e6a4983b5<br>---<br>M configure.ac<br>M src/Makefile.am<br>M src/conv_acc.c<br>A src/conv_acc_neon.c<br>A src/conv_acc_neon_impl.h<br>5 files changed, 508 insertions(+), 0 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/configure.ac b/configure.ac</span><br><span>index f69c78d..2397b2f 100644</span><br><span>--- a/configure.ac</span><br><span>+++ b/configure.ac</span><br><span>@@ -378,6 +378,17 @@</span><br><span> AM_CONDITIONAL(HAVE_SSE4_1, false)</span><br><span> fi</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+AC_ARG_ENABLE(neon,</span><br><span style="color: hsl(120, 100%, 40%);">+ [AS_HELP_STRING(</span><br><span style="color: hsl(120, 100%, 40%);">+ [--enable-neon],</span><br><span style="color: hsl(120, 100%, 40%);">+ [Enable NEON support]</span><br><span style="color: hsl(120, 100%, 40%);">+ )],</span><br><span style="color: hsl(120, 100%, 40%);">+ [neon=$enableval], [neon="no"])</span><br><span style="color: hsl(120, 100%, 40%);">+AC_DEFINE(HAVE_NEON,,</span><br><span style="color: hsl(120, 100%, 40%);">+[Support ARM NEON instructions])</span><br><span style="color: hsl(120, 100%, 40%);">+AM_CONDITIONAL(HAVE_NEON, [test "x$neon" != "xno"])</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> OSMO_AC_CODE_COVERAGE</span><br><span> </span><br><span> dnl Check if the compiler supports specified GCC's built-in function</span><br><span>diff --git a/src/Makefile.am b/src/Makefile.am</span><br><span>index 16119d9..be09784 100644</span><br><span>--- a/src/Makefile.am</span><br><span>+++ b/src/Makefile.am</span><br><span>@@ -48,6 +48,11 @@</span><br><span> endif</span><br><span> endif</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+if HAVE_NEON</span><br><span style="color: hsl(120, 100%, 40%);">+libosmocore_la_SOURCES += conv_acc_neon.c</span><br><span style="color: hsl(120, 100%, 40%);">+# conv_acc_neon.lo : AM_CFLAGS += -mfpu=neon no, could as well be vfp with neon</span><br><span style="color: hsl(120, 100%, 40%);">+endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> BUILT_SOURCES = crc8gen.c crc16gen.c crc32gen.c crc64gen.c</span><br><span> EXTRA_DIST = conv_acc_sse_impl.h crcXXgen.c.tpl</span><br><span> </span><br><span>diff --git a/src/conv_acc.c b/src/conv_acc.c</span><br><span>index c16e436..0f6f7ca 100644</span><br><span>--- a/src/conv_acc.c</span><br><span>+++ b/src/conv_acc.c</span><br><span>@@ -85,6 +85,11 @@</span><br><span> void osmo_conv_sse_avx_vdec_free(int16_t *ptr);</span><br><span> #endif</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+#ifdef HAVE_NEON</span><br><span style="color: hsl(120, 100%, 40%);">+int16_t *osmo_conv_neon_vdec_malloc(size_t n);</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_vdec_free(int16_t *ptr);</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> /* Forward Metric Units */</span><br><span> void osmo_conv_gen_metrics_k5_n2(const int8_t *seq, const int16_t *out,</span><br><span> int16_t *sums, int16_t *paths, int norm);</span><br><span>@@ -129,6 +134,21 @@</span><br><span> int16_t *sums, int16_t *paths, int norm);</span><br><span> #endif</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+#if defined(HAVE_NEON)</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k5_n2(const int8_t *seq, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm);</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k5_n3(const int8_t *seq, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm);</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k5_n4(const int8_t *seq, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm);</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k7_n2(const int8_t *seq, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm);</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k7_n3(const int8_t *seq, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm);</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k7_n4(const int8_t *seq, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm);</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> /* Trellis State</span><br><span> * state - Internal lshift register value</span><br><span> * prev - Register values of previous 0 and 1 states</span><br><span>@@ -528,6 +548,12 @@</span><br><span> if (dec->k == 5) {</span><br><span> switch (dec->n) {</span><br><span> case 2:</span><br><span style="color: hsl(120, 100%, 40%);">+/* rach len 14 is too short for neon */</span><br><span style="color: hsl(120, 100%, 40%);">+#ifdef HAVE_NEON</span><br><span style="color: hsl(120, 100%, 40%);">+ if (code->len < 100)</span><br><span style="color: hsl(120, 100%, 40%);">+ dec->metric_func = osmo_conv_gen_metrics_k5_n2;</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span> dec->metric_func = osmo_conv_metrics_k5_n2;</span><br><span> break;</span><br><span> case 3:</span><br><span>@@ -681,6 +707,8 @@</span><br><span> } else {</span><br><span> INIT_POINTERS(gen);</span><br><span> }</span><br><span style="color: hsl(120, 100%, 40%);">+#elif defined(HAVE_NEON)</span><br><span style="color: hsl(120, 100%, 40%);">+ INIT_POINTERS(neon);</span><br><span> #else</span><br><span> INIT_POINTERS(gen);</span><br><span> #endif</span><br><span>diff --git a/src/conv_acc_neon.c b/src/conv_acc_neon.c</span><br><span>new file mode 100644</span><br><span>index 0000000..7244946</span><br><span>--- /dev/null</span><br><span>+++ b/src/conv_acc_neon.c</span><br><span>@@ -0,0 +1,110 @@</span><br><span style="color: hsl(120, 100%, 40%);">+/*! \file conv_acc_neon.c</span><br><span style="color: hsl(120, 100%, 40%);">+ * Accelerated Viterbi decoder implementation</span><br><span style="color: hsl(120, 100%, 40%);">+ * for architectures with only NEON available. */</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * (C) 2020 by sysmocom - s.f.m.c. GmbH</span><br><span style="color: hsl(120, 100%, 40%);">+ * Author: Eric Wild</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * All Rights Reserved</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * SPDX-License-Identifier: GPL-2.0+</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is free software; you can redistribute it and/or modify</span><br><span style="color: hsl(120, 100%, 40%);">+ * it under the terms of the GNU General Public License as published by</span><br><span style="color: hsl(120, 100%, 40%);">+ * the Free Software Foundation; either version 2 of the License, or</span><br><span style="color: hsl(120, 100%, 40%);">+ * (at your option) any later version.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is distributed in the hope that it will be useful,</span><br><span style="color: hsl(120, 100%, 40%);">+ * but WITHOUT ANY WARRANTY; without even the implied warranty of</span><br><span style="color: hsl(120, 100%, 40%);">+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span><br><span style="color: hsl(120, 100%, 40%);">+ * GNU General Public License for more details.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * You should have received a copy of the GNU General Public License along</span><br><span style="color: hsl(120, 100%, 40%);">+ * with this program; if not, write to the Free Software Foundation, Inc.,</span><br><span style="color: hsl(120, 100%, 40%);">+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#include <stdlib.h></span><br><span style="color: hsl(120, 100%, 40%);">+#include <stdint.h></span><br><span style="color: hsl(120, 100%, 40%);">+#include <malloc.h></span><br><span style="color: hsl(120, 100%, 40%);">+#include "config.h"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#if defined(HAVE_NEON)</span><br><span style="color: hsl(120, 100%, 40%);">+#include <arm_neon.h></span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* align req is 16 on android because google was confused, 8 on sane platforms */</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_ALIGN 8</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#include <conv_acc_neon_impl.h></span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* Aligned Memory Allocator</span><br><span style="color: hsl(120, 100%, 40%);">+ * NEON requires 8-byte memory alignment. We store relevant trellis values</span><br><span style="color: hsl(120, 100%, 40%);">+ * (accumulated sums, outputs, and path decisions) as 16 bit signed integers</span><br><span style="color: hsl(120, 100%, 40%);">+ * so the allocated memory is casted as such.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+int16_t *osmo_conv_neon_vdec_malloc(size_t n)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ return (int16_t *) memalign(NEON_ALIGN, sizeof(int16_t) * n);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_vdec_free(int16_t *ptr)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ free(ptr);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k5_n2(const int8_t *val, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const int16_t _val[4] = { val[0], val[1], val[0], val[1] };</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ _neon_metrics_k5_n2(_val, out, sums, paths, norm);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k5_n3(const int8_t *val, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const int16_t _val[4] = { val[0], val[1], val[2], 0 };</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ _neon_metrics_k5_n4(_val, out, sums, paths, norm);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k5_n4(const int8_t *val, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const int16_t _val[4] = { val[0], val[1], val[2], val[3] };</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ _neon_metrics_k5_n4(_val, out, sums, paths, norm);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k7_n2(const int8_t *val, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const int16_t _val[4] = { val[0], val[1], val[0], val[1] };</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ _neon_metrics_k7_n2(_val, out, sums, paths, norm);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k7_n3(const int8_t *val, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const int16_t _val[4] = { val[0], val[1], val[2], 0 };</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ _neon_metrics_k7_n4(_val, out, sums, paths, norm);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__attribute__ ((visibility("hidden")))</span><br><span style="color: hsl(120, 100%, 40%);">+void osmo_conv_neon_metrics_k7_n4(const int8_t *val, const int16_t *out,</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *sums, int16_t *paths, int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const int16_t _val[4] = { val[0], val[1], val[2], val[3] };</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ _neon_metrics_k7_n4(_val, out, sums, paths, norm);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span>diff --git a/src/conv_acc_neon_impl.h b/src/conv_acc_neon_impl.h</span><br><span>new file mode 100644</span><br><span>index 0000000..4471127</span><br><span>--- /dev/null</span><br><span>+++ b/src/conv_acc_neon_impl.h</span><br><span>@@ -0,0 +1,354 @@</span><br><span style="color: hsl(120, 100%, 40%);">+/*! \file conv_acc_neon_impl.h</span><br><span style="color: hsl(120, 100%, 40%);">+ * Accelerated Viterbi decoder implementation:</span><br><span style="color: hsl(120, 100%, 40%);">+ * straight port of SSE to NEON based on Tom Tsous work */</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * (C) 2020 by sysmocom - s.f.m.c. GmbH</span><br><span style="color: hsl(120, 100%, 40%);">+ * Author: Eric Wild</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * All Rights Reserved</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * SPDX-License-Identifier: GPL-2.0+</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is free software; you can redistribute it and/or modify</span><br><span style="color: hsl(120, 100%, 40%);">+ * it under the terms of the GNU General Public License as published by</span><br><span style="color: hsl(120, 100%, 40%);">+ * the Free Software Foundation; either version 2 of the License, or</span><br><span style="color: hsl(120, 100%, 40%);">+ * (at your option) any later version.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This program is distributed in the hope that it will be useful,</span><br><span style="color: hsl(120, 100%, 40%);">+ * but WITHOUT ANY WARRANTY; without even the implied warranty of</span><br><span style="color: hsl(120, 100%, 40%);">+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span><br><span style="color: hsl(120, 100%, 40%);">+ * GNU General Public License for more details.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * You should have received a copy of the GNU General Public License along</span><br><span style="color: hsl(120, 100%, 40%);">+ * with this program; if not, write to the Free Software Foundation, Inc.,</span><br><span style="color: hsl(120, 100%, 40%);">+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/* Some distributions (notably Alpine Linux) for some strange reason</span><br><span style="color: hsl(120, 100%, 40%);">+ * don't have this #define */</span><br><span style="color: hsl(120, 100%, 40%);">+#ifndef __always_inline</span><br><span style="color: hsl(120, 100%, 40%);">+#define __always_inline inline __attribute__((always_inline))</span><br><span style="color: hsl(120, 100%, 40%);">+#endif</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_BUTTERFLY(M0,M1,M2,M3,M4) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ M3 = vqaddq_s16(M0, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M4 = vqsubq_s16(M1, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M0 = vqsubq_s16(M0, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M1 = vqaddq_s16(M1, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = vmaxq_s16(M3, M4); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M3 = vreinterpretq_s16_u16(vcgtq_s16(M3, M4)); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M4 = vmaxq_s16(M0, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M1 = vreinterpretq_s16_u16(vcgtq_s16(M0, M1)); \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_DEINTERLEAVE_K5(M0,M1,M2,M3) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8x2_t tmp; \</span><br><span style="color: hsl(120, 100%, 40%);">+ tmp = vuzpq_s16(M0, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = tmp.val[0]; \</span><br><span style="color: hsl(120, 100%, 40%);">+ M3 = tmp.val[1]; \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_DEINTERLEAVE_K7(M0,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10,M11,M12,M13,M14,M15) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8x2_t tmp; \</span><br><span style="color: hsl(120, 100%, 40%);">+ tmp = vuzpq_s16(M0, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M8 = tmp.val[0]; M9 = tmp.val[1]; \</span><br><span style="color: hsl(120, 100%, 40%);">+ tmp = vuzpq_s16(M2, M3); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M10 = tmp.val[0]; M11 = tmp.val[1]; \</span><br><span style="color: hsl(120, 100%, 40%);">+ tmp = vuzpq_s16(M4, M5); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M12 = tmp.val[0]; M13 = tmp.val[1]; \</span><br><span style="color: hsl(120, 100%, 40%);">+ tmp = vuzpq_s16(M6, M7); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M14 = tmp.val[0]; M15 = tmp.val[1]; \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_BRANCH_METRIC_N2(M0,M1,M2,M3,M4,M6,M7) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ M0 = vmulq_s16(M4, M0); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M1 = vmulq_s16(M4, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = vmulq_s16(M4, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M3 = vmulq_s16(M4, M3); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M6 = vcombine_s16(vpadd_s16(vget_low_s16(M0), vget_high_s16(M0)), vpadd_s16(vget_low_s16(M1), vget_high_s16(M1))); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M7 = vcombine_s16(vpadd_s16(vget_low_s16(M2), vget_high_s16(M2)), vpadd_s16(vget_low_s16(M3), vget_high_s16(M3))); \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_BRANCH_METRIC_N4(M0,M1,M2,M3,M4,M5) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ M0 = vmulq_s16(M4, M0); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M1 = vmulq_s16(M4, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = vmulq_s16(M4, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M3 = vmulq_s16(M4, M3); \</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t t1 = vpadd_s16(vpadd_s16(vget_low_s16(M0), vget_high_s16(M0)), vpadd_s16(vget_low_s16(M1), vget_high_s16(M1))); \</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t t2 = vpadd_s16(vpadd_s16(vget_low_s16(M2), vget_high_s16(M2)), vpadd_s16(vget_low_s16(M3), vget_high_s16(M3))); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M5 = vcombine_s16(t1, t2); \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_NORMALIZE_K5(M0,M1,M2,M3) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = vminq_s16(M0, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t t = vpmin_s16(vget_low_s16(M2), vget_high_s16(M2)); \</span><br><span style="color: hsl(120, 100%, 40%);">+ t = vpmin_s16(t, t); \</span><br><span style="color: hsl(120, 100%, 40%);">+ t = vpmin_s16(t, t); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = vdupq_lane_s16(t, 0); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M0 = vqsubq_s16(M0, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M1 = vqsubq_s16(M1, M2); \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define NEON_NORMALIZE_K7(M0,M1,M2,M3,M4,M5,M6,M7,M8,M9,M10,M11) \</span><br><span style="color: hsl(120, 100%, 40%);">+{ \</span><br><span style="color: hsl(120, 100%, 40%);">+ M8 = vminq_s16(M0, M1); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M9 = vminq_s16(M2, M3); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M10 = vminq_s16(M4, M5); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M11 = vminq_s16(M6, M7); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M8 = vminq_s16(M8, M9); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M10 = vminq_s16(M10, M11); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M8 = vminq_s16(M8, M10); \</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t t = vpmin_s16(vget_low_s16(M8), vget_high_s16(M8)); \</span><br><span style="color: hsl(120, 100%, 40%);">+ t = vpmin_s16(t, t); \</span><br><span style="color: hsl(120, 100%, 40%);">+ t = vpmin_s16(t, t); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M8 = vdupq_lane_s16(t, 0); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M0 = vqsubq_s16(M0, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M1 = vqsubq_s16(M1, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M2 = vqsubq_s16(M2, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M3 = vqsubq_s16(M3, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M4 = vqsubq_s16(M4, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M5 = vqsubq_s16(M5, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M6 = vqsubq_s16(M6, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+ M7 = vqsubq_s16(M7, M8); \</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__always_inline void _neon_metrics_k5_n2(const int16_t *val, const int16_t *outa, int16_t *sumsa, int16_t *paths,</span><br><span style="color: hsl(120, 100%, 40%);">+ int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict out = __builtin_assume_aligned(outa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict sums = __builtin_assume_aligned(sumsa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8_t m0, m1, m2, m3, m4, m5, m6;</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t input;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and expand 8-bit input out to 16-bits */</span><br><span style="color: hsl(120, 100%, 40%);">+ input = vld1_s16(val);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vcombine_s16(input, input);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and compute branch metrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vmulq_s16(m2, m0);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vmulq_s16(m2, m1);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vcombine_s16(vpadd_s16(vget_low_s16(m0), vget_high_s16(m0)),</span><br><span style="color: hsl(120, 100%, 40%);">+ vpadd_s16(vget_low_s16(m1), vget_high_s16(m1)));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Load accumulated path matrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&sums[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&sums[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_DEINTERLEAVE_K5(m0, m1, m3, m4)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Butterflies: 0-7 */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m3, m4, m2, m5, m6)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (norm)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_NORMALIZE_K5(m2, m6, m0, m1)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[0], m2);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[8], m6);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[0], m5);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[8], m4);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__always_inline void _neon_metrics_k5_n4(const int16_t *val, const int16_t *outa, int16_t *sumsa, int16_t *paths,</span><br><span style="color: hsl(120, 100%, 40%);">+ int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict out = __builtin_assume_aligned(outa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict sums = __builtin_assume_aligned(sumsa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8_t m0, m1, m2, m3, m4, m5, m6;</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t input;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and expand 8-bit input out to 16-bits */</span><br><span style="color: hsl(120, 100%, 40%);">+ input = vld1_s16(val);</span><br><span style="color: hsl(120, 100%, 40%);">+ m4 = vcombine_s16(input, input);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and compute branch metrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[16]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[24]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N4(m0, m1, m2, m3, m4, m2)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Load accumulated path matrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&sums[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&sums[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_DEINTERLEAVE_K5(m0, m1, m3, m4)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Butterflies: 0-7 */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m3, m4, m2, m5, m6)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (norm)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_NORMALIZE_K5(m2, m6, m0, m1)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[0], m2);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[8], m6);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[0], m5);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[8], m4);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__always_inline static void _neon_metrics_k7_n2(const int16_t *val, const int16_t *outa, int16_t *sumsa, int16_t *paths,</span><br><span style="color: hsl(120, 100%, 40%);">+ int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict out = __builtin_assume_aligned(outa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict sums = __builtin_assume_aligned(sumsa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8_t m0, m1, m2, m3, m4, m5, m6, m7;</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8_t m8, m9, m10, m11, m12, m13, m14, m15;</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t input;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Load accumulated path matrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&sums[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&sums[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&sums[16]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&sums[24]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m4 = vld1q_s16(&sums[32]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m5 = vld1q_s16(&sums[40]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m6 = vld1q_s16(&sums[48]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m7 = vld1q_s16(&sums[56]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Deinterleave into even and odd packed registers */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_DEINTERLEAVE_K7(m0, m1, m2, m3, m4, m5, m6, m7, m8, m9, m10, m11, m12, m13, m14, m15)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and expand 8-bit input out to 16-bits */</span><br><span style="color: hsl(120, 100%, 40%);">+ input = vld1_s16(val);</span><br><span style="color: hsl(120, 100%, 40%);">+ m7 = vcombine_s16(input, input);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and compute branch metrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[16]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[24]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N2(m0, m1, m2, m3, m7, m4, m5)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[32]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[40]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[48]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[56]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N2(m0, m1, m2, m3, m7, m6, m7)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Butterflies: 0-15 */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m8, m9, m4, m0, m1)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m10, m11, m5, m2, m3)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[0], m0);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[8], m2);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[32], m9);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[40], m11);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Butterflies: 17-31 */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m12, m13, m6, m0, m2)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m14, m15, m7, m9, m11)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[16], m0);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[24], m9);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[48], m13);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[56], m15);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (norm)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_NORMALIZE_K7(m4, m1, m5, m3, m6, m2, m7, m11, m0, m8, m9, m10)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[0], m4);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[8], m5);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[16], m6);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[24], m7);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[32], m1);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[40], m3);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[48], m2);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[56], m11);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+__always_inline static void _neon_metrics_k7_n4(const int16_t *val, const int16_t *outa, int16_t *sumsa, int16_t *paths,</span><br><span style="color: hsl(120, 100%, 40%);">+ int norm)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict out = __builtin_assume_aligned(outa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16_t *__restrict sums = __builtin_assume_aligned(sumsa, 8);</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8_t m0, m1, m2, m3, m4, m5, m6, m7;</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x8_t m8, m9, m10, m11, m12, m13, m14, m15;</span><br><span style="color: hsl(120, 100%, 40%);">+ int16x4_t input;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Load accumulated path matrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&sums[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&sums[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&sums[16]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&sums[24]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m4 = vld1q_s16(&sums[32]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m5 = vld1q_s16(&sums[40]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m6 = vld1q_s16(&sums[48]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m7 = vld1q_s16(&sums[56]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Deinterleave into even and odd packed registers */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_DEINTERLEAVE_K7(m0, m1, m2, m3, m4, m5, m6, m7, m8, m9, m10, m11, m12, m13, m14, m15)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and expand 8-bit input out to 16-bits */</span><br><span style="color: hsl(120, 100%, 40%);">+ input = vld1_s16(val);</span><br><span style="color: hsl(120, 100%, 40%);">+ m7 = vcombine_s16(input, input);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (BMU) Load and compute branch metrics */</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[0]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[8]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[16]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[24]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N4(m0, m1, m2, m3, m7, m4)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[32]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[40]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[48]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[56]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N4(m0, m1, m2, m3, m7, m5)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[64]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[72]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[80]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[88]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N4(m0, m1, m2, m3, m7, m6)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ m0 = vld1q_s16(&out[96]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m1 = vld1q_s16(&out[104]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m2 = vld1q_s16(&out[112]);</span><br><span style="color: hsl(120, 100%, 40%);">+ m3 = vld1q_s16(&out[120]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BRANCH_METRIC_N4(m0, m1, m2, m3, m7, m7)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Butterflies: 0-15 */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m8, m9, m4, m0, m1)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m10, m11, m5, m2, m3)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[0], m0);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[8], m2);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[32], m9);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[40], m11);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* (PMU) Butterflies: 17-31 */</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m12, m13, m6, m0, m2)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_BUTTERFLY(m14, m15, m7, m9, m11)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[16], m0);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[24], m9);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[48], m13);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&paths[56], m15);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (norm)</span><br><span style="color: hsl(120, 100%, 40%);">+ NEON_NORMALIZE_K7(m4, m1, m5, m3, m6, m2, m7, m11, m0, m8, m9, m10)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[0], m4);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[8], m5);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[16], m6);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[24], m7);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[32], m1);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[40], m3);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[48], m2);</span><br><span style="color: hsl(120, 100%, 40%);">+ vst1q_s16(&sums[56], m11);</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/libosmocore/+/19372">change 19372</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/libosmocore/+/19372"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: libosmocore </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I58ff2cb4ce3514f43390ff0a2121f81e6a4983b5 </div>
<div style="display:none"> Gerrit-Change-Number: 19372 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: Hoernchen <ewild@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Hoernchen <ewild@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: fixeria <vyanitskiy@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: laforge <laforge@osmocom.org> </div>
<div style="display:none"> Gerrit-CC: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>