Dear Jonathan,
without knowing the details, it may very well be that you are hitting resourec allocation limits on the USB bus. It is a shared bus, after all. Every additional device will require bandwidth; every endpoint on every device will need to be periodically polled according to its descriptors.
The USB host software (OHCI/UICH/EHCI) or USB host controller firmware (XHCI) are managing this, and with hard real time sensitive streaming of SDR samples, you cannot afford to significantly delay any of them.
It's best to use USB ports connected to other USB root hubs / controllers for your additional devices.