Setting IRQ affinity for network controllers to either follow or not follow the "high priority" processes seems to do the trick.

I usually do stuff described here:

https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.tx...

And sometimes following ends up better (same cache), sometimes isolating ends up being better.