Truenas remote replication fails/disconnects after 30m when transferring at near line speed
Both of us are on att fiber/1gbit. I'm bypassed on gpon. Remote client sits behind a bgw320, also on gpon. Connection to all remote endpoints is over wg tunnel.
Had a zfs replication task fail at exactly 30m - it was a single snapshot that had ~180GB of new data since the last snapshot. Most of it completed. This was a daily scheduled overnight replication task in TN. Replication task is configured using ssh+nc.
This morning i ran a simple single stream iperf3 test to the same target. It too failed at 30m (30m20s to be exact).
I then reran the same iperf3 test from the same local client to my vps endpoint. No issues after 30m. Speeds were more sporadic given it's a vps.
From my the local truenas instance to remote tn I get consistent 890mbps upload over wg. Test to the vps test fluctuated between 400 and 700 mbps.
Note, after both failures, it took 6-7 min to re-establish the wg connection. Remote logs indicated no internet connectivity at the remote end. It could ping the default gateway ip (192.168.1.254), but not the internet and no dns (server configured at 192.168.1.254) resolution.
Pings to 192.168.1.254 were successful indicating the bgw did not reboot. Also, bgw's uptime does not indicate a crash.
An additional iperf3 test performed, this time limiting upload bandwidth to 500 mbps (-b 500M) was successful. I then tried another one at full speed with same 30m failure.
I don't have much remote access to the bgw other than to ask the person running it to check certain things. Given the test to vps was successful suggests that att has some limits implemented on the bgw. I do recall some other replication tasks in the past that moved even more data (600gb) successfully, but that was scattered over multiple datasets and snapshots. I don't believe there was a single snapshot exceeding 100GB.
Is this a known limitation with att or is there some other factor at play?