Update on the $15 Cavium SNIC10E 10GbE card: prebuilt image, second-host benchmarks, and I need beta testers
A month ago I posted about turning the Cavium CN6640-SNIC10E into a working dual-port 10GbE card with no vendor firmware. Here's what changed since, plus a clarification and a request.
WHAT'S NEW
- Prebuilt card image in the releases. No more building OpenWrt for hours: download the image, run install.sh.
- The host module now installs through DKMS, so it survives kernel upgrades.
- Card temperatures show up in sensors with no serial cable, plus an estimated power draw.
- Rewritten quick start, one block from git clone to a working oct0. Hardcoded paths are gone.
- Requirements fixed: "Above 4G decoding" is NOT needed. Secure Boot has to be off.
SECOND-HOST RESULTS
Same card re-measured on a different machine and peer (Dell R720 + HP NC523): 9.64 Gb/s TX and 9.93 Gb/s RX aggregate across both ports, matching the first rig. Each port alone reaches line rate, but the two together share about 10 Gb/s per direction, because the card sits on a PCIe Gen2 x4 link.
A CLARIFICATION ON "NOTHING FLASHED"
The card OS always runs from RAM and is never written to flash. There are two ways to boot it:
- Over serial (boot-clean.sh): nothing is written to the card at all. Power it off and it's exactly as it was.
- Serial-free (the default): a one-time step saves the u-boot environment to NAND, so the host can boot the card on its own. That is a write, but restore-bootapp.sh puts the stock env back.
So "fully reversible" holds either way, and "nothing flashed" is strictly true only for the serial path. The docs now say this plainly.
KNOWN ISSUES
- Heavy TX collapses RX under full duplex. Still hunting the cause.
- ARP broadcasts sometimes don't leave the card after a reload (static neighbour entries work around it).
- Hot-plugging a DAC can leave TX stuck until the service is restarted.
BETA TESTERS WANTED
So far this has only run on my own card. If you have an SNIC10E in a drawer, I'd love to know if it works on yours. You need a Linux host with Secure Boot off and a USB-serial adapter for the one-time provisioning. Useful feedback: host model, kernel version, whether install.sh worked first try, and iperf3 numbers.
Report here, even just to say "it worked": https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic/discussions/1
Repo: https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic
Docs: https://nicologiuliani.site/docs/cavium-cn6640-snic10e/
Write-up: https://nicologiuliani.site/blog/cavium-cn6640-bar2-datapath/
SMALL SPOILER
After the first post someone reached out asking if the same approach could open up a Sophos XGS 107w firewall. I've been working on it and it's almost done. More soon.
If you want to support the work, GitHub Sponsors is here: https://github.com/sponsors/nicologiuliani6