r/ShittySysadmin

Cursed idea:

​PAM module that using an LLM with the power to overrule your sudo privledges.

Example scenarios:

• "Why do you need root?"
• "Why are you on a Vietnamese IP?"
• "Convince me of your identity."

To the more cursed:

• "Okay, but first solve this riddle...."
• "Public key checks out, but it's 3am. It's time to go to bed."
• "Nginx has been running for 71 days now, I think ​it deserves a nap."
• "Keystroke dynamics suggest you are drunk. Sudo denied."​

Well - it happened. The day started like any other, except my colleague said their email got hacked. I looked, and indeed it was hacked. Immediately, I reset his password. Then another person said their email got hacked too. I asked both of them when the last time they changed their password was: never.

I wish I was making this up. I called our email provider and asked them if they could let me know who was still on the default. Turns out it was 1/3 of our organization. My predecessor never instituted 'force user to change password', and then nobody changed it. The password for all those emails was 'password'.

This is the same guy that didn't realize you need device isolation nor network isolation for a public wifi network. I guess I shouldn't be surprised. I seriously wish I were making this up.

I've clicked it 12 times and made direct eye contact with the laptop.

The deployment still says Pending. Does it need to see my manager?

I listen to the color of the servers. Its become 'my thing' on my team. Whenever something goes wrong, i go into the DC and slowly walk among the racks, absorbing what they have to tell me.

The NAS and the switch are my brothers.

How far does this network cable go? If you cut it, then well never know

Documentation is technical debt with punctuation.

Think about it: if a system is properly configured, why would it need documentation? The server already knows how it works. The firewall knows what ports are open. DNS knows where everything lives. The only person who apparently doesn’t know is "Kevin," and frankly that sounds like a Kevin problem.

Documentation also creates a dangerous single point of truth. The moment you write down how something works, somebody will read it six months later and assume it is still accurate. Now you have two problems: the actual infrastructure and a fictional universe where the infrastructure still runs the last gold star release firmware or OS and not X.0.1...

By refusing to document anything, we eliminate stale documentation entirely. Our documentation accuracy rate becomes perfect because zero percent of zero documents are incorrect.

There are security benefits too. Hackers can’t steal your network diagrams if the network diagrams exist only as memories spread across three burned-out sysadmins. That’s basically human RAID 5 with one failing drive.

And consider job security. If anyone can follow a runbook and restart any system successfully, what exactly are they paying you for? Institutional knowledge should be treated like an encryption key: kept in one person’s head, never backed up, and taken on vacation without notice.

Some people call this a “bus factor of one.” I call it aggressive knowledge centralization.

Besides, writing documentation takes time away from important sysadmin work like investigating why disk usage went from 40% to 97% overnight, renewing certificates that expire tomorrow, and staring at a terminal while saying, “That’s weird.”

If future employees need to understand the environment, they can learn it the same way we did: SSH in, run commands they found on Stack Overflow, and gradually develop an intuitive understanding of the system through fear.

Documentation doesn’t build character.

Outages build character.