I did some more tests. I upgraded the OS on my Odroid XU4 to Ubuntu 18.04, and this has libusb 1.0.21 in the repo's (the previous one I was using had libusb 1.0.20). Now I get the same continuous lost sample bytes problem on the XU4 as on the Tinkerboard - a huge amount of dropped samples with zerocopy enabled.
However, on the latest Raspbian on the Rpi3, I don't see the problem. On the Pi3 rtl_test returns only one line of dropped samples, then no more.
Allocating 15 zero-copy buffers
lost at least 156 bytes
On the Odroid and Tinkerboard it's more like:
Allocating 15 zero-copy buffers
lost at least 13617588 bytes
lost at least 12326518 bytes
lost at least 13208366 bytes
lost at least 13301044 bytes
...and so on forever
No other error messages are seen, and disabling the usbfs_memory_mb limit by setting it to zero doesn't get rid of the problem.
I'm not too sure about the CMA stuff, i'm just using the default OS images provided for the Tinkerboard and XU4, and standard Ubuntu 18.04 on my laptop.