Looking for help on jails networking
Hello,
I've been trying to setup a handful of jails to get into freebsd for a student project.
Using VNET is rather straightforward, but I'm getting into an issue when I try to chain a few things together, and maybe I'm just taking it the wrong way.
For the project, I end up with the setup you can see on bsdrp, without the HA part, which isn't in the cards for now.
One of the issue here is having a private network for a specific environment. Because I could simply list a bunch of vnet interfaces into a single firewall/relayd jail but it also mean bringing down the jail if I want to add or remove a jail from private network and clean up. So I've been looking into using bridges for the private network.
It'd mean ending up with: bridge0 -> server firewall -> dedicated firewall -> dedicated bridge -> dedicated jails.
That way, we can just add on the bridge0 a new epair for a new dedicated environment, spin a dedicated bridge and firewall jail and we have done most of the work to setup some dedicated environment.
The goal is to get something like this:
┌─────────────┐
│host-level pf│
└──────┬──────┘
│
│ ┌────xn1
│ │
▼ │
wan──────────bridge0────┤
│
│ ┌────────────────────┐ ┌─────────┐
├──epairA───────┤private-env-level pf├───bridgeA──┬──epairAX──────┼webserver│
│ └────────────────────┘ │ └─────────┘
│ │
│ │ ┌─────────┐
│ ├──epairAY──────┤database │
│ │ └─────────┘
│ │
│ │ ┌─────────┐
│ └──epairAZ──────┤ dns │
│ └─────────┘
│
│ ┌────────────────────┐ ┌─────────┐
├──epairB───────┤private-env-level pf├───bridgeB──┬──epairBX──────┼webserver│
│ └────────────────────┘ │ └─────────┘
│ │
. │ ┌─────────┐
. ├──epairBY──────┤database │
. │ └─────────┘
│
│ ┌─────────┐
└──epairBZ──────┤ dns │
└─────────┘
My Questions:
Is it worth it to do it like this? I feel like scrapping the second bridge make managing new jails more painful. What setup would help if I want to redo this project later on?
I've an issue, my jails behind the second bridge can reach everybody within the private network.
I've set the gateway of the private jails to the IP the epair of the private-env-level pf has on the private bridge. When using netstat -r to check it out, it is extremely slowly writing out the proper routes, but it takes a long time, more than 30 seconds
# netstat -r
Routing tables
Internet:
Destination Gateway Flags Netif Expire
I'm bad at networking, but IIRC the gateway should be the next jump in the network.
So for example: My firewall jail has 2 IPs:
- 10.10.10.10/24 as WAN.
- 172.16.1.1/27 as LAN.
My private jail should have the LAN as gateway, isn't it? Or I'm wrong and it explain the slowness of the netstat -r.
I've looked into it a bit, and it seems removing the LRO option is important, I've checked and turned that off, but no improvement. Does anyone have pointers there?
Concerning the NATs Should I have a double NAT (private-env-level pf + host-level pf) ? The host-level pf one is fine, I've used it before and had no issue with it. The other one is a bit more out there for me: With a setup using bridge0 as public bridge. and bridge6 as private bridge. I've gathered that I should do this:
# Macros
ext_if = "bridge0"
lan_if = "bridge6"
nat on $lan_if from <jails> to any -> ($ext_if)
But it doesn't work. Anybody got input on that?
One of the goal here is to stick to plain jails, ifconfig and such, forcing myself to get low-level enough I've to learn the concepts. I've also excluded VLANs as of now (the multiple vlans on a single bridge in FreeBSD15 seems really nice), as it'd require changes to my hypervisor, which seems a bit finicky, mostly because its documentation doesn't cover my use case, and it is a student project with a deadline, so I've to stick to a rather simple setup. I'd be very happy to get back to the project once the deadline is past to start getting netgraph and akin in, but as of now, I'm trying to get something working for the deadline.
I'd be very glad for any kind of documentation that can walk me through that because as I pointed out, I'm not a network person, I'm kind of a dumbass when it comes to network.