I tried calibrating LO leakage cancellation again and found out that it wasn't calibrated enough. The settings are very sensitive and increasing the I and Q shifts by one made a large difference on the E4406. The RSA seems to compensate, though, because the same changes don't affect it at all.
I also tried some different frequencies. At the not useful frequency of 240 MHz, we are not far from target values. At 945 MHz, it is better than before, but still too high. The 1900 MHz band is not so good.
Attached measurements are all at 4 sps, with the updated linear pulse from yesterday. I played more with the CP setting; the value of 0x93 provided the best measurements.
Thomas