▲ 30 r/HomeDataCenter+1 crossposts

Update on the $15 Cavium SNIC10E 10GbE card: prebuilt image, second-host benchmarks, and I need beta testers

A month ago I posted about turning the Cavium CN6640-SNIC10E into a working dual-port 10GbE card with no vendor firmware. Here's what changed since, plus a clarification and a request.

WHAT'S NEW

- Prebuilt card image in the releases. No more building OpenWrt for hours: download the image, run install.sh.
- The host module now installs through DKMS, so it survives kernel upgrades.
- Card temperatures show up in sensors with no serial cable, plus an estimated power draw.
- Rewritten quick start, one block from git clone to a working oct0. Hardcoded paths are gone.
- Requirements fixed: "Above 4G decoding" is NOT needed. Secure Boot has to be off.

SECOND-HOST RESULTS

Same card re-measured on a different machine and peer (Dell R720 + HP NC523): 9.64 Gb/s TX and 9.93 Gb/s RX aggregate across both ports, matching the first rig. Each port alone reaches line rate, but the two together share about 10 Gb/s per direction, because the card sits on a PCIe Gen2 x4 link.

A CLARIFICATION ON "NOTHING FLASHED"

The card OS always runs from RAM and is never written to flash. There are two ways to boot it:

- Over serial (boot-clean.sh): nothing is written to the card at all. Power it off and it's exactly as it was.
- Serial-free (the default): a one-time step saves the u-boot environment to NAND, so the host can boot the card on its own. That is a write, but restore-bootapp.sh puts the stock env back.

So "fully reversible" holds either way, and "nothing flashed" is strictly true only for the serial path. The docs now say this plainly.

KNOWN ISSUES

- Heavy TX collapses RX under full duplex. Still hunting the cause.
- ARP broadcasts sometimes don't leave the card after a reload (static neighbour entries work around it).
- Hot-plugging a DAC can leave TX stuck until the service is restarted.

BETA TESTERS WANTED

So far this has only run on my own card. If you have an SNIC10E in a drawer, I'd love to know if it works on yours. You need a Linux host with Secure Boot off and a USB-serial adapter for the one-time provisioning. Useful feedback: host model, kernel version, whether install.sh worked first try, and iperf3 numbers.

Report here, even just to say "it worked": https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic/discussions/1

Repo: https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic

Docs: https://nicologiuliani.site/docs/cavium-cn6640-snic10e/

Write-up: https://nicologiuliani.site/blog/cavium-cn6640-bar2-datapath/

SMALL SPOILER

After the first post someone reached out asking if the same approach could open up a Sophos XGS 107w firewall. I've been working on it and it's almost done. More soon.

If you want to support the work, GitHub Sponsors is here: https://github.com/sponsors/nicologiuliani6

reddit.com
u/Shiny1712 — 12 days ago
▲ 160 r/firetvstick+2 crossposts

Tracked down a Fire TV Stick flooding the LAN with ~300 mDNS pkt/s, killing all Wi-Fi. Root cause, an NSEC responder, and on-device removal

This one took me a while so I figured I'd write it up. Every AP in the house died at once, 46 to 60% packet loss on wireless, while wired gigabit didn't blink. So airtime, not bandwidth.

Ran a tcpdump on port 5353:

sudo tcpdump -i <iface> -nn -e -c 3000 port 5353

One Fire TV Stick was firing AAAA queries at ~180/sec, which lands as ~300 pps of multicast on the segment. Multicast goes out at the lowest common rate to everyone, so it eats 2.4GHz airtime for the whole network.

Turns out the Stick keeps asking for the AAAA of a *.fritz.box name, and the FRITZ!Box answers with an empty packet, 0 questions 0 records. Fire OS doesn't count that as an answer or as silence, so it never backs off and just retries forever. Amazon knows about it, it was reported 2026-07-28 and it's still open.
What each fix did. At baseline it was ~300 pps and 46 to 60% loss. Killing the offending device brought it to ~1 pps and 0% loss. With the NSEC responder running it sat at ~0.1 pps and 0% loss.

The responder is a little stdlib-only Python thing. It answers the query with a proper negative reply (NSEC, "this name has no AAAA") plus the A record on the end, so the Stick negative-caches it (120s TTL) and shuts up. 12 replies killed 881 queries. Runs as a systemd unit, or procd on OpenWrt.

If you'd rather fix it at the source: temporary root with a build-pinned kernel exploit (CVE-2026-43499, no bootloader unlock, gone after a reboot), then pm uninstall the three WhisperLink packages that do the querying. The removal sticks across reboots even though root doesn't. After that: 3 mDNS packets in 15s, none from the Fire TV.

Full write-up with the whole script, the systemd unit, the stuff that didn't work, the kara vs karat codename gotcha, and the minimal fix I sent Amazon:

https://nicologiuliani.site/blog/firetv-mdns-storm/

u/Shiny1712 — 16 days ago
▲ 141 r/openwrt

Turned a $15 eBay "smart NIC" nobody could get working into a real 10GbE card (no vendor firmware, fully reversible)

The Cavium CN6640-SNIC10E is a dual-port 10GbE PCIe card that shows up on eBay for ~$15–20. It's an OEM "SmartNIC," an 8-core Octeon II SoC meant to run Cavium's vendor LiquidIO firmware. Problem: the stock liquidio driver basically doesn't work with these OEM boards (its flash-handshake boot model doesn't match), and secondhand cards tend to have bootloaders in a rough state. There's a good writeup of people hitting a wall on this back in 2021 on the OpenWrt forums. Both PCIe BARs would read back all-0xFF and the board looked architecturally dead as anything but LiquidIO.

Turns out that "BAR wall" is just unprogrammed PEM inbound registers, not a hardware limit. You can program them from the card side with plain u-boot write64 (no firmware needed) and open a full bidirectional PCIe channel.

On top of that I built:

- an OpenWrt image that runs on the card from RAM (nothing flashed, fully reversible)

- a reverse-engineered PCIe BAR2 shared-memory datapath between host and card

- an out-of-tree host kernel module (octnic) that registers the card as two independent Linux NICs, oct0/oct1

Result: line-rate 10GbE, TX at 9.7–9.8 Gb/s (zero-copy), RX at 8.1–8.8 Gb/s (DMA via the Octeon DPI engine), both ports live simultaneously on a DAC. Boots with no serial cable after a one-time provisioning step. A systemd service brings both NICs up automatically at boot, so day to day it's just a normal 10GbE card as far as the rest of the homelab is concerned.

No vendor NDA firmware anywhere in the stack, host OS untouched, and the whole thing is undoable if you want the card back to stock.

Repo (driver, boot tooling, docs): https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic

Docs cover architecture, first-time flashing/provisioning over serial, usage, and full performance numbers if anyone wants to reproduce this on their own card.

Happy to answer questions. This took a fair bit of reverse engineering to get past the BAR wall so ask away if you're stuck on one of these boards.

If you want to help out (test on your own board, contribute code, whatever) open an issue or discussion on the repo. If you'd like to support the project, GitHub Sponsors is enabled there too.

UPDATE: new idea, Another interesting use case is running OpenWrt standalone directly on the card.

For around $13, you basically get a dual-port 10GbE SFP+ router, without needing a separate x86 machine. And with two or more of these, you could potentially build a pretty cheap multi-router failover setup.

That's actually one of the things I'd like to explore next.

u/Shiny1712 — 1 month ago
▲ 1.2k r/HomeDataCenter+3 crossposts

Turned a $15 eBay "smart NIC" nobody could get working into a real 10GbE card (no vendor firmware, fully reversible)

The Cavium CN6640-SNIC10E is a dual-port 10GbE PCIe card that shows up on eBay for ~$15–20. It's an OEM "SmartNIC," an 8-core Octeon II SoC meant to run Cavium's vendor LiquidIO firmware. Problem: the stock liquidio driver basically doesn't work with these OEM boards (its flash-handshake boot model doesn't match), and secondhand cards tend to have bootloaders in a rough state. There's a good writeup of people hitting a wall on this back in 2021 on the OpenWrt forums. Both PCIe BARs would read back all-0xFF and the board looked architecturally dead as anything but LiquidIO.

Turns out that "BAR wall" is just unprogrammed PEM inbound registers, not a hardware limit. You can program them from the card side with plain u-boot write64 (no firmware needed) and open a full bidirectional PCIe channel.

On top of that I built:

- an OpenWrt image that runs on the card from RAM (nothing flashed, fully reversible)

- a reverse-engineered PCIe BAR2 shared-memory datapath between host and card

- an out-of-tree host kernel module (octnic) that registers the card as two independent Linux NICs, oct0/oct1

Result: line-rate 10GbE, TX at 9.7–9.8 Gb/s (zero-copy), RX at 8.1–8.8 Gb/s (DMA via the Octeon DPI engine), both ports live simultaneously on a DAC. Boots with no serial cable after a one-time provisioning step. A systemd service brings both NICs up automatically at boot, so day to day it's just a normal 10GbE card as far as the rest of the homelab is concerned.

No vendor NDA firmware anywhere in the stack, host OS untouched, and the whole thing is undoable if you want the card back to stock.

Repo (driver, boot tooling, docs): https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic

Docs cover architecture, first-time flashing/provisioning over serial, usage, and full performance numbers if anyone wants to reproduce this on their own card.

Happy to answer questions. This took a fair bit of reverse engineering to get past the BAR wall so ask away if you're stuck on one of these boards.

If you want to help out (test on your own board, contribute code, whatever) open an issue or discussion on the repo. If you'd like to support the project, GitHub Sponsors is enabled there too. https://github.com/sponsors/nicologiuliani6

u/Shiny1712 — 1 month ago