
Odd Packet Loss Behavior w/ Intel X540 Under Load (Unifi Dream Machine Pro Involved)
I'm hoping to get some help on this one, it's very odd, and very specific. I'll give as much necessary detail as I can, but if I were to just dump it all I think this post would get too long for anyone to want to read it lol.
I recently made a change from a Netgate 6100 running pfSense to a Unifi Dream Machine Pro in my network (already had it around) due to it's faster routing performance. After doing so, most things are working really well, but I started noticing a packet loss issue on my desktop when it's under heavy load.
Basic topology for reference:
pfSense was only capable of around 2 gigabit downloads via Steam or Epic Games, whereas this can do much higher (depends on the game, in my testing we're looking at about 2.5 gigabit).
Summary of issue while downloading a Steam game at roughly 2.2-2.5 Gbps:
- Pinging Google results in 1/30 or so packets lost with 10-20ms std dev
- Pinging a LAN IP results in less, but still present (maybe 1/60) packet loss and basically no std dev
- Ping Plotter showed a few % packet loss for each hop going to Google
- UDMP Htop shows it's NOT under maximum load, and since this happens on Layer 2 as well, I don't think it's the culprit
- Pinging Google from a VM on the XCP-ng host connected to the same switch results in zero packet loss and almost no std dev
- This leads me to believe Layer 2 and the UDMP Firewall are working as intended and not the culprits
What's really odd about all this is iPerf3 testing does not produce the same symptoms. If I ping Google or a LAN IP, while doing inter-VLAN routed iPerf3 testing I can sustain 8-9 Gbps without a single lost packet (even running for 10 minutes straight).
During the inter-VLAN testing the UDMP Htop shows 100% on all cores, unlike the Steam download. I would imagine if the UDMP was the culprit here I would see the issue present the higher the load.
At first, I thought maybe this issue stemmed from NAT or state table overflow, but I'm not so confident in that now that the VM can ping externally without a single lost packet.
Additionally, I don't see this issue happen when running high bandwidth (6 gigabit +) speed tests from my PC to the internet.
I'm starting to think either A. something is going on with my NIC or driver causing an issue, or maybe even the cabling. But if that was the case, wouldn't it also be present with iPerf3 and SpeedTest? Or B. maybe it's related to the UDMP and it's ability to handle states since Steam will use multiple connections. It's just odd to me that the Layer 2 ping to a local LAN IP also breaks apart, the UDMP, at that point, isn't involved at all. So I'd assume it has to be related to the switch if that was the case.