automated config changes to OPNsense box

Hi all, i'm working on automating some stuff within my home lab, primarily through automated nix deployments.

one small roadblock i currently have towards my goal of being able to take a nix config fully from a configuration on my workstation to a running set of services on a VM on my proxmox cluster is that my homelab is actually closer to 6 homelabs, experimenting with various things (i actually use my homelab as a lab to learn new skills, not just to make storing my "linux iso's" or w/e easier).

sadly, im stuck with my main residential router, which doesn't support basic features like using a larger subnet, using multiple subnets or using VLANs.

so currently, i am running a double-NAT setup with an opnsense box behind my main router, my "home net" is on the main network, and the opnsense box effectively condenses my entire home lab down to one ip.

this *fully* works, i have it set up to work completely fine, though it took some doing. learnt a *lot* about NAT and firewalls trying to get everything working correctly especially since my proxy is on the main net with multiple newt vpns on various subnets to reach the different servers that need public internet publishing, which all has to work "locally" but through NAT.

i've already gotten my deployment pipeline to the point where the server is created with the desired configuration and subsequently moved on proxmox to the correct network/vlan (all servers are prepped by booting an ISO that has a predetermined IP on the local net, since deployment of the config happens over ssh).

problem is: once this happens, i no longer have access to the server over ssh because now it's on a different subnet behind NAT.

this is easily resolved by opening a port on the opnsense router, i already took this into accounts so all my configs have a port noted in case they need to go through the opnsense router.

the actual thing i cannot seem to figure out is: how do i set up a NAT destination rule and its accompanying firewall rule automatically as part of a deployment pipeline?

it's totally fine for this router to have a bunch of ssh ports open btw, since well... its behind *another* firewall that basically only has http, https, email and VPN ports available, all bound to specific servers. there's no way to reach any of these ports from the internet.

basically: what's the best approach to having an automated workflow of some kind make changes to the NAT configuration/firewall configuration? preferably something that doesn't need its own server to run on, but if i must, i will.

if this is not possible at all and someone knows of another way to automatically gain ssh access from my main workstation to all deployed servers on the various opnsense subnets (assuming those servers are already set up with proper keys, sshd configs etc), do also let me know about those.

reddit.com
u/SDG_Den — 1 day ago
▲ 164 r/NixOS

Github is down, so nixpkgs with flakes does not work. How can i avoid this in the future?

currently, in my flake i'm using github:nixos/nixpkgs/nixos-unstable for my nixpkgs.url, this means that when github is down, I cannot update my system and generally cannot make certain changes to my config.

are there any good ways to set this up with some form of failover? I'd like to not be fully reliant on github considering their uptime is an absolute joke for a site of that scope (99.22% right now!)

reddit.com
u/SDG_Den — 3 days ago
▲ 15 r/NixOS

New(*) nixOS power-user, would like some resources on how to do specific things.

Hi All, im an IT professional and hobbyist, and i've been looking into nixOS for a project.

To prep and learn nixOS, ive switched my desktop over and started by just making a well thought-out config. Currently, im using a flake-parts based config, which has 3 hosts on it atm (my laptop, my desktop and my partner's desktop, who's diving into nixOS alongside me). I've integrated home manager and most of my system is managed declaratively.

I also have a homelab consisting of 4 HPE servers (total 128c 660gb ram 30tb storage), theyre running proxmox.

My eventual goal is to design and build a central management system for nixOS endpoints, one thing that has bothered me to no end is that there is nothing like microsoft's intune/entra ID stack for linux, and i think the way nixOS works makes it a prime candidate for building a proof-of-concept around it.

My current goal is to expand my config and create a toolchain for deploying new servers to my proxmox hosts and keep their configs up to date. Every server will be a host in my current multi-host config, and my idea is to use nixos-anywhere and disko to deploy and the remote functionality to update configs.

To get servers to the point where i can access them with nixos-anywhere, my plan is to build a nixos live ISO. Im not *sure* if nixos-anywhere can function off of a liveboot, but considering it bootstraps nixOS and completely wipes any installed system, i'd guess a liveboot would work fine.

Then i can tie it together by using SSH and scripting to automatically create the VM, place the iso on the right server, start it and run nixos-anywhere to deploy the desired config.

I'd *like* to automate it further, to where when i run my gitupdate command (which has some extra logic to build the commit message automatically already) it'll rebuild the configuration of any servers that have changed.

One thing i've had some trouble with is finding resources. So i'd like to ask if anyone here has any resources i can read through for these specific questions:

- 1. Is there a built-in way using scripting to figure out if a specific system configuration in your flake has changed? If not, how could i best approach this?

- 2. The goal of the eventual centralized management project is to be able to fully manage remote enterprise machines and have a centralized user directory. What is currently considered the best way to handle centralized user directories on nixOS? Currently my biggest thought is to just... Use nix. If the config is central and the users are configured declaratively, then the users are centrally managed.

- 3. How would you approach deploying a config across many systems regardless of which network they may be on? Think laptops. The two main options im currently considering are a declaratively configured VPN server/tunnel and automation on the server to push configs to endpoints over the tunnel, and using a private git host with a private repository using ssh key authentication, which would require the endpoints to poll for changes to their config somehow. One is push-based, one is pull-based. The big thing is that the git option could be "serverless".

- 4. Are there any good resources on docker on nixOS? or should i switch to a different containerization approach? Im relatively familiar with docker.

- 5. For the tooling, i'll likely just write some bash scripts and wrap them as services where needed. Are there any good guides out there on how to make a derivation for groups of bash scripts? (Entrypoint + libraries + needed files). And how do i make custom services on nixOS?

- 6. What's the best way to handle SSH hardening for my home lab?

- 7. What are some best practices when it comes to keeping such a many-system config workable and maintainable? What about the deployment pipelines?

- 8. Is there any danger with importing a flake-parts module multiple times or setting the same package as installed in multiple files?

If you have any knowledge on these topics or any links you can throw my way, please share them!

reddit.com
u/SDG_Den — 3 days ago

Tell me your worst Linux Sin and I will tell you your cruel and unusual punishment.

as the title says: confess in the comments what your worst linux sin is, and you will be dealt a cruel, unusual and (hopefully) funny punishment. this is an all-judgement zone, feel free to judge.

reddit.com
u/SDG_Den — 9 days ago
▲ 36 r/homelab

Custom server management utility

a bit of a crossover: I really like building my own tooling, and recently, i built this!

this is a TUI/cli system for managing my servers built on zellij and fzf, it gives me a main window with all my sessions (using a stacked layout with the menu on the right) and a second window with options to add, remove, edit, move between categories or even inject the server-side components.

this is running over hardened ssh (key-only), and has cli/tui options for executing commands on a remote server or on an entire group or even a specific selection at once.

on the server side, it opens another zellij window with a terminal + yazi for file browsing, lazydocker, btop and a split showing live firewall status and port activity.

the little window in the bottom left shows basic information about the device including a live-check for the server-side components as well as fetching the OS live.

since this is a terminal utility, it can be used over SSH itself and remain fully interactive, including mouse functionality! everything is clickable.

to add to this, i actually also have a package manager and build system that i use for all of my little utilities, and i've extended them today to also be able to handle containers, so with my upcoming projects, i'll first be building container packages with my own build system that can be deployed using my own package manager via my own server management interface. it's all just simple bash scripting but its honestly very nice to be able to create your own automation like this!

this is not really a project showcase in the sense of "i built this cool project, please use it". this is *my* server manager tailored exactly to *my* needs. if you think this is cool, go look into how to make your own little CLI and TUI tools via bash scripting! it's surprisingly easy and pretty fun to do, most of my tools are effectively just wrappers around existing tools that make them work more easily for what i want to do. you do not need to be a programmer to do this, just know the basics of using a CLI.

u/SDG_Den — 25 days ago

In light of all the recent distro-posting: why use one distro? why not all of them?

finally, i have them all. apt, dnf, xbps, pacman, yay, apk, portage and zypper.

if you want to know how: i hijacked a cachyOS install with bedrock. it's been working pretty damn well so far outside of root snapshots breaking.

u/SDG_Den — 1 month ago

Bedrock on CachyOS

Hi all, just wanted to drop by to report i just hijacked a semi-fresh install of cachyOS (reinstalled my thinkpad, re-deployed my dotfiles, then hijacked with bedrock), and it *just worked*.

I didn't expect to run into many issues considering arch seems pretty well supported, but i was expecting some more difficulty with the BTRFS+limine setup.

is there anything i have to watch out for when it comes to that + snapshots via snapper? how does that work exactly?

reddit.com
u/SDG_Den — 1 month ago

the latest linux video "it's time to commit to linux, but which one?" - Some feedback from a linux nerd.

Hi all, I just finished watching the new linux video (here), and honestly? I thought it was pretty good!

I have a pretty long list of things that new users should know before picking a distribution, and while short, LTT managed to mention *basically all of them*.

That being said, here's my nitpicks. Keep in mind, everything else was great!

distro choice: While it makes sense to test the more "gaming-specific" distro's (which are generally tier 2 or tier 3 distributions like cachyOS or bazzite), I think it'd have been good to also include some tier 1 choices (Fedora, Debian, Arch, openSUSE). The main reason for this is that it's generally recommended for a new user to stick close to upstream since those distro's have the biggest userbase and thus the largest amount of people online able to help answer any questions you may have online.

software availability: LTT mentions this as "third-party app support", and while it was mentioned, I hope there will be a follow-up on this topic covering in more details what the differences are in both software availability and release cadence between different distributions. For example, one thing they have not noticed in their video at all is that ubuntu, fedora and something like cachyOS all update at different speeds due to their release filosophy. This is because all of the distributions they've chosen have their own more up-to-date repositories for gaming software, which *is* a good reason to pick one of these distro's over something like ubuntu desktop or fedora KDE.

speaking of desktops: Desktop Environments. I don't think LTT did this topic justice in the video. There are so many ways you could configure your desktop I do believe this, too, deserves its own video talking about the "big" desktops (KDE, GNOME, XFCE, cinnamon, budgie, cosmic) as well as window managers (hyprland, sway, i3, niri etc) and how they differ.

Lastly, the big one in my opinion is the complete absence of any talk about what an atomic/immutable distribution is. The reason why this is important is because of their final recommendation: Bazzite or CachyOS.

Bazzite is an immutable distribution, while CachyOS is not.

this means that when running Bazzite, the core OS is read-only and cannot be changed. Modifying or troubleshooting your install is harder and you have to install any side-applications you want via something like distrobox.

in exchange for this, Bazzite is incredibly unlikely to break. This makes Bazzite an obvious choice for things like handhelds and livingroom consoles, where all you really do on them is open steam and play games.

However, it's a much more frustrating experience to tinker with than something like CachyOS, which due to being arch-based can be really easily modified while keeping it safe to do so through pre-configured snapshots.

picking between these two distro's is honestly down to how much you *want* to tinker, and i'd have liked for LTT to have mentioned that in the video.

I hope that we'll see more videos about these topics in the future for a more detailed look! It'd also be interesting to see LTT do a video on how to distrohop, since there's a couple things you can do to make it much easier on yourself.

oh btw, my actual recommendation for a distro is: *whatever you feel works best for you*. a lot of setups are incredibly opinionated and people have differing opinions.

u/SDG_Den — 1 month ago

all they do is click through a GUI every day, they have no idea how things actually work or what happened when something breaks, they just know which button fixes what issue.

is this just the companies i've worked for or do windows-exclusive sysadmins just have skill issues?

also i would like to thank all 7 pixels for coming together for this post. it was hard work but we got it done.

u/SDG_Den — 1 month ago

Plebeians click through a GUI, a real Arch Wizard commands their system directly.

before anyone asks, i got bored again.

so next to my previous glyph-based input, i've decided to make fully programmable voice input using whisper AI.

it can run commands, open terminals, open programs and type text, including carrying over the remainder of the activation sentence.

for example, the next sentence was typed entirely using this tool:

arch is the best

u/SDG_Den — 2 months ago

all these novices with their "tab to complete", a real ARCH wizard *casts* his full system upgrades.

i got bored again and made this.

using my wacom tablet to draw/write, which is why the mouse jitters a little.

project not-so-proudly slop-coded in python using GTK libraries.

of course, it has customizability, you can add your own glyphs and it has a text mode that.... "works".

u/SDG_Den — 2 months ago

ALL PRAISE BE TO THE CORPORATE MICROSLOP OVERLORDS!

no, this is *not* fedora pretending to be azure linux, i actually went through the trouble of downloading the azure linux 4.0 ISO, installing it, figuring out how to get KDE plasma running on it by jankily adding the fedora 43 repositories from scratch and then installing it and fastfetch.

of course, fastfetch has no logo for it out-of-the-box, but i'm sure that can be resolved.

and yes, the azure linux logo being in the bottom left *was* set up by default, i did not configure shit. this is literally the first launch of plasma.

small update: this guy spent some extra time ricing theirs: https://www.reddit.com/r/linuxmemes/comments/1ulqx05/they_said_it_couldnt_be_done/

u/SDG_Den — 2 months ago

Got bored and made a dynamic UI framework for my WM setup, what could I use it for?

(this is not an advertisement, more-so a brainstorming post)

I recently built myself a simple bash-based framework to add dynamic UI elements on top of my window manager, currently it can trigger in 3 ways:

-#1: the mouse enters/leaves a specified zone

-#2: the layout on a specific monitor changes to/from a specific layout

-#3: a specific program gets focused/unfocused

it has per-mode delays built in, and the triggers just run commands so technically, i can wire it into anything.

so far, i've added a volume bar on the right of my screen that pops up when you move your mouse there, a brightness bar on the left and a screenshot utility at the top, all of these using zones.

i've also added a tab bar that only shows in monocle and deck modes (Where windows are effectively hiding behind eachother and it's normally impossible to tell how many windows you have open and in which order)

next to UI, this framework can start/stop basically anything or even just *launch an initialization script* when a condition is detected. So basically:

what can you think of that would be useful to have automatically start/stop or show/hide based on where on the screen the mouse is, what program is focused or what layout the window manager is on?

particularly interested in what the focus/unfocus trigger could be used for, i'm sure there's something but there's nothing obvious within my own workflow yet.

reddit.com
u/SDG_Den — 2 months ago
▲ 41 r/kde

An update on my really stupid project (wayland WM + KDE plasma shell)

in my previous post on this sub, i asked for some help trying to get plasma to work with a different WM, this used to be possible on x11 but is a lot harder on wayland.

I am happy to let you all know i have succeeded.... to a degree.

what works:

- bottom bar and launcher work, you can launch applications and applets

- settings menu from KDE (mostly) works

- all wayfire functionality works

- background settings from KDE shell work

- desktop icons from KDE shell work

what doesn't work (as far as i've been able to tell)

- any of the built-in KDE keybinds

- the power/logoff/restart buttons in the KDE launcher

i'm also still getting some flickering issues and issues with windows spawning on the wrong workspace, the flickering may be entirely unrelated to KDE (the joys of using an nvidia card with wayland WMs) and the windows spawning on the wrong workspace is possibly a quirk of how the KDE plasma launcher menu interacts with wayfire's "multi-desktop" situation, my multiple desktops are laid out in a grid that i can zoom out to, and all programs spawn near the center of that 3x3 grid (so around workspace 5)

not really a good user-experience atm, but hey! it's "functional".

what i had to do:

- symlink /etc/xdg/menus/plasma-applications.menu to /etc/xdg/menus/applications.menu as well as to ~/.config/applications.menu

- start both wayfire *and* plasmashell with dbus-launch, with plasmashell being started from wayfire

- start the following from wayfire using autostart: kded6, ksmserver, xdg-desktop-portal, a polkit agent (and a terminal, but that's optional, just useful for troubleshooting)

i should be able to fix at least some of these issues, nevertheless, it shouldn't have to be said that YOU SHOULD NOT DO THIS WITH THE EXPECTATION OF HAVING A STABLE OR FUNCTIONAL SYSTEM.

this is very much a situation where i'm asking myself whether or not i could, not whether or not i should. I'm a professional Idiot in Tech, and i have at least 3 other desktops to rely on if this one doesnt work, plus this isn't my main desktop. this is just for fun, and until a more official way of swapping your WM is provided, should serve only as a concept for what that would be like *at best*

edit: i have now also tested this with *mangoWM*, which is what i normally use, and got the same results just by turning off my DankMaterialShell and running plasmashell instead. you will still need to do the symlinking to get your applications to show (but i already did that for my wayfire test)

on mangoWM, i do not get any flickering, so that's likely an issue with nvidia + wayfire.

u/SDG_Den — 2 months ago

common linux pitfals and mistakes for newbies

I've been working on some more guides since some people have asked for that, and i'd like to include sections on both linux pitfalls and common mistakes.

I'll be defining pitfalls as "problems a new user will frequently run into", problems can be something being broken or just something not working the way a new user (coming from mac or windows) would expect.

i'll be defining mistakes as things a new user may do on accident (for example, accidentally running some kind of `sudo rm -rf /` is a popular one)

I'd love to hear both about pitfalls/mistakes you've seen others run into, as well as pitfalls/mistakes you've personally seen or made!

(and yall, please don't get judgy in the comments, i'm pretty sure everyone will eventually make a stupid mistake. i personally managed to run `sudo rm -rf $DIR/*` with DIR being empty, effectively deleting my entire drive. that was thankfully 2 days in so it wasnt too bad.)

reddit.com
u/SDG_Den — 2 months ago

What are some common pitfalls and mistakes for new linux users?

I've been working on some more guides since some people have asked for that, and i'd like to include sections on both linux pitfalls and common mistakes.

I'll be defining pitfalls as "problems a new user will frequently run into", problems can be something being broken or just something not working the way a new user (coming from mac or windows) would expect.

i'll be defining mistakes as things a new user may do on accident (for example, accidentally running some kind of `sudo rm -rf /` is a popular one)

I'd love to hear both about pitfalls/mistakes you've seen others run into, as well as pitfalls/mistakes you've personally seen or made!

(and yall, please don't get judgy in the comments, i'm pretty sure everyone will eventually make a stupid mistake. i personally managed to run `sudo rm -rf $DIR/*` with DIR being empty, effectively deleting my entire drive. that was thankfully 2 days in so it wasnt too bad.)

reddit.com
u/SDG_Den — 2 months ago

Non openSUSE user with some questions

Hi Everyone!

as the title says, I do not personally use openSUSE, I've worked with it *a bit* before, but nothing major.

Recently I wrote a guide on r/linux4noobs about picking a distro, which went down pretty well, so I'm planning on writing up more educational guides for linux in general.

However, that guide did not include openSUSE *at all*. It primarily focuses on Fedora, Arch and Ubuntu (and all of the distributions based on them).

I'd *like* to learn a bit more about openSUSE, as I do think it should be included into guides like these.

Of course, I will be making a VM with openSUSE once more, this time primarily to explore the distro, but I also have some questions for the community!

Why are you (currently) on openSUSE and do you plan to stick around?

How do you feel openSUSE LEAP compares to other big "versioned release" distro's like Ubuntu and Fedora?

How do you feel openSUSE tumbleweed compares to other "rolling release" distro's like arch and void?

What do you think openSUSE does different/better than other popular distributions?

What kind of users do you feel openSUSE is for?

I primarily want to understand what the actual users think, rather than just *my* opinion, as distro choice is a personal matter and different options suit different users.

Would love to hear from you all and have a nice day!

reddit.com
u/SDG_Den — 2 months ago
▲ 6 r/kde

Need some insight and help for a *really* stupid project involving KDE plasma

hold onto your hats, this one may require a little explaining.

So, i am not a KDE user. My partner is, but i'm not.

We both use arch (by the way), and i personally do a lot of tinkering around with window managers as a hobby, building different isolated dotfiles around different window managers that i can access all at the same time, that do not interact with eachother (meaning that for each set, i use a different wallpaper manager, a different shell, a different terminal, the only things that stay the same are applications that are not part of the dotfiles, like discord and such). sometimes i publish these.

I found out that with the x11 version of KDE, it is possible to replace KWin with a different window manager, the arch wiki has a guide on this for i3.

thing is, i don't want to use an x11 window manager, I'd like to see if i can get the plasma shell working fully with both Wayfire (floating/stacking, like kwin) and MangoWM (scrolling, autotiling, master/stack, monocle etc etc, basically every auto-tiling layout under the sun) using a separate desktop file for each (so they can exist alongside a normal KDE plasma install without messing it up)

So far, with a little experimentation, i've made a desktop session that uses a custom init script which loads both wayfire and the plasma shell directly, however, it seems running JUST plasma-shell is mildly broken.

the way you set a different WM on plasma x11 sadly doesn't work on wayland, unless i'm misunderstanding how to get it to run.

so now, i need to basically know either A: how to run start-plasma-wayland with a custom WM instead of KWin or B: what other components need to be launched for the plasma shell to work properly so i can manually launch them all using the custom init script.

I'm *specifically* looking to tinker around with plasma with different wayland window managers, i'm not looking to try out Karousel or Khronkite or make other modifications to KWin. I get that as a practical solution, those are likely a lot easier and safer than what i'm doing, but this won't be my daily driver and it's honestly mostly to try stuff and have fun doing it. we're not going for practical, we're going for "can it be done", followed by "can i make it reliable/stable" if the answer is "yes".

i'm not afraid of tinkering around and writing custom scripting/configs, my whole system has proper backups and snapshots so if i do bung something up, i can recover without any problems and i'm pretty familiar with these kinds of things, just not KDE specifically.

reddit.com
u/SDG_Den — 2 months ago

[META] i recently posted a guide on how to pick a distro, and quite liked writing it. what other guides for linux newbies would be much appreciated?

Hi everyone, i'm Den, i'm an IT professional and general linux nerd. I've also spent the last 5 years guiding people through their first destiny raids.

I've been spending the last while answering questions here and in r/linuxquestions, and eventually, i got tired of repeating the same advice about picking a distro over and over again so i made this guide.

I was thinking, what other topics would be helpful to have a guide on for new users? i'd imagine proton and proton troubleshooting would be a good one, maybe the basics of docker so people can use containers, a dual-booting guide, what else?

whether you're a newbie struggling to find info on a topic or a 10+ year linux veteran that happens to know about a great topic that newbies should know, sound off in the replies!

reddit.com
u/SDG_Den — 2 months ago