u/naptastic

The narrow end of my Ethernet fabric is two Ubiquiti US-24's from 2018. They have two gigabit SFP uplinks each. The switch they're connected to is a Mellanox SX6036, using a breakout cable to turn a 40gbE port into four 1gbE ports. Bit of a waste if you ask me.

But then I check eBay for US-48's, which have four SFP+ uplinks, and the first two pages of results are "parts only / for repair" and those aren't even POE models. YIKES. No more Ubiquiti for me then.

What good switches can break out a 40gbE connection (or multiple SFP+ connections) into twenty-four or more 1gbE RJ45 ports?

Here's the behavior I want:

• For packets going switch-to-switch (SFP ports):
  • If a VLAN tag is present, pass the packet on.
  • If no VLAN tag is present, drop the packet.
• For packets going host-to-switch (RJ-45 ports):
  • If no VLAN tag is present, the switch adds a tag with that port's VID.
  • If a VLAN tag is present and VID=0, the switch should force the VID to that port's VID.
  • If a VLAN tag is present and VID is any other value, the switch should drop the packet. (Even if it's tagged for that port's VLAN.)
• For packets going switch-to-host (same RJ-45 ports):
  • VLAN tags get stripped, so the packet arriving at the host does not have an 802.1q tag at all.

I've never actually used VLANs before, but I thought this is what "trunk" and "access" meant? The Ubiquiti controller doesn't seem to have such concepts. I can configure a port to have a "native" VLAN and either accept or block traffic tagged for other VLANs, but it always passes "Default" traffic.

I think. Actually, I don't know what it's doing.

The interface is opaque and has many fewer options than it did when I bought the switches in 2019. It just wants me to create a cloud account, install an app on my phone, and buy more hardware and licenses. NO, NO, and NO.

Wtihout typing out the whole saga of what I've tried, all roads lead to network loops and broadcast storms.

How do I get this behavior out of a UniFi US-24, or should I sell them and buy a different brand, or am I fundamentally misunderstanding what VLANs are for and how to use them?

(Background: I'm used to running an Infiniband-only underlay. I would create VXLAN overlays and bridge those to physical ports on quad-gig cards. This works, but it requires hosts to be on all the time, and I'm trying to save on electricity. Also, the VXLAN overlays are done in the CPU, and the UniFi switches don't support configuring ports as VTEPs. How much nicer that would be...)

[SOLVED]: There were two subnet managers running.

Sometimes my Infiniband interfaces don't come up, and I see this error in dmesg. `ibstat` says State: Active, Physical state: LinkUp, rate 10. (Rate should be 56.) The switch (Mellanox SX6036) gives the same information.

I've tried OpenSM as provided by Debian (do not use), OpenSM 5.20.0 from MLNX-OFED, and the subnet manager built into the SX6036, which is on the latest firmware.

I have seen this error condition on every single HCA in the fabric at some point:

• ConnectX-3 FCBT
• Connect-IB
• ConnectX-4 FCAT
• ib0 inside my SX6036 switch, which is on the latest firmware

The fabric inspector inside the switch does not see anything connected to the Infiniband fabric.

I have also used an SX6005, which does not have the embedded CPU, so there's no dmesg to check, and it's never been a problem.

I've never disabled multicast. IPoIB works, VXLAN overlays work, SRP works, iSCSI works, NFS/RDMA works... except in hosts with this error condition.

There are enough PCIe resources in the hosts; I can lower the amounts requested by the HCA arbitrarily and nothing changes. I can turn off SR-IOV and sometimes it fixes things the error stops, but usually not. Sometimes a full cold boot resolves it, but usually not.

There's no way I'm running out of multicast groups; I have exactly one IB partition, and only 5 hosts connected to it.

Please advise?