r/sysadmin

Hey fellow sys admin,

I work at a VFX/animation studio (~400 users).

We're kicking off a project to migrate our artist workstations from Windows 10/11 VDIs to Linux. Flavor is pretty much locked it ll be Rocky Linux 9 with KDE.

I am curious if there's anyone here who's been through a similar workstation migration to linux?

What Linux flavor, did you end up on, and would you pick the same again?

How did you handle what GPOs used to give you?

For the Base OS deployment/imaging, did you use Kickstart + Puppet/Ansible, or something else?

Anything that looked easy on paper and turned into a one big pile of {Jurassic_Park}?

Tools that made your life easier along the way?

I am looking for any war stories, good or bad!

Thank you!

I got a call from one of our rural customers at this MSP and their coaxial internet was down again. They lose like $1000+ an hour if they're down so they told me they called Verizon, their business cell provider, and asked if they had something. I assume a salesman mentioned it in the past. After one phone call where I said "our networking guy is out today but I'm pretty sure you can plug just about anything into the WAN port on the Fortigate and it'll work" they called back an hour later saying they got an a XC46BE. I've never even heard of that family of devices so I thought this is gonna be a shit day.

Luckily for my non-networking specialist self, was easier to set up than a 54g linksys in 2002. Basically it just jumps on and spits out wifi and ethernet. I ran a test on my non-verizon smartphone and got 16mbps to the tower. Then I hooked my laptop up to the device and got 220x40 so fuck net neutrality I guess. It even has a battery and their switches had UPSes so that's a bit interesting.

I slapped it into the Fortigate and tada, everyone's back online...with about 10mbps and 500ms ping time. I assumed all their Outlooks were syncing at once or something but they told her the device can do about 20 people. They have about 10 highly active computers. Pretty unimpressive for $350! But they intended to return it when the outage was fixed and their rep said that was fine.

The very millisecond I walked out the door, the ISP truck showed up and started messing with the box on the front lawn. Awesome use of my time.

But I keep hearing about these magical devices that can switch over to 5G and we do have WAN1 and WAN2 on the Fortigate so they're considering keeping it and programming in a switchover of some sort. I assume they do that. Anyone have one that doesn't suck? Because this one impressed me until it was actually in use. I saw one at a trade show years ago that was a UPS + 5G modem. That sounded kinda neat but so did this Verizon device until it performed poorly.

I'm revisiting an effort I did about a year ago. I'm looking for a better way. I want to find a process that will parse current user sessions on a server (active/disconnected/idle/ect.) and log the accounts off if their username matches a string ("adm_").

I'd love to find an off the shelf solution rather than have to support a homebrew PowerShell solution.

Give me what you have, even if it is an alternate PowerShell/scripting option. Something has to be better than the nightmare my script turned into.

Genuinely asking

• A 15–20-year-old system may still be doing its job perfectly.
• Replacing it can introduce cost, migration risks, downtime, and employee training.
• But legacy systems can become difficult to maintain and integrate with modern APIs, cloud services, mobile apps, and etc.
• At what point does it still works become a business risk?
• Is modernization/migration better than completely rebuilding from scratch?

So I've started seeing that SSL Lifespan is shortening - going down to eventually supposed to be every 47 days.

We're a small shop, but we have a lot of different services. I've been doing my best when I have free time to catalog everything that has an SSL Cert, but I know I'm missing stuff.

I've seen a bit about ACME Clients and such; and from what I've heard it's great. They handle rotating the certs and all.

But something for me just isn't clicking. For instance, we have a lot of large scale copiers, ala your Ricoh or Lexmark or Brother. We have those locked down with SSL Certs, but we have to manually push those up to it.

Now as these are internal services that aren't externally facing, I don't see no reason why we can't self-issue those certs; but currently our CTO likes to utilize a paid for Wildcard for all our internal stuff.

I keep quite busy so haven't had too much time to really dig in on researching, but I know the time bomb is ticking.

So for those who are managing SSL Certs and all, and potentially utilizing ACME Clients and such, what should I expect and whats the general gist of what my workflow should be?

I was wondering how many of you are still using legacy client/server apps, and come across challenges when it comes to remote/hybrid workers.

We use an application that is rather old school. No web based access, full desktop client and server. It does not work well under any latency, so VPN's and anything similar are unworkable.

The other challenge is that the application is heavily tied into MS Word, Outlook, etc. The application uses COM addins for Outlook to generate and send emails and allow interaction with the product. Similar happens with MS word.

We've tried publishing the application as a "remote app" on platforms like Citrix, but the issue again is the required integrations with Outlook, Word, downloading of files like PDF's and other documents that are stored within the app. It essentially integrates heavily with other apps.

So at the moment, we're using a full desktop experience for users over Citrix. I find this to be rather frustrating. Providing end users with new laptops with reasonable spec, for them to work remotely and use it like a thin client so they can connect to a Citrix desktop. It confuses users, it isn't particularly user friendly, and it just feels like a waste of local resources.

Does anyone work in similar situations? Are there any solutions that could help with this? Or are we destined to forever use a full remote desktop solution to support our core, oldschool, clunky-ass product?

Hi guys, we had a client whom had a hostile takeover of their 365 admin portal who assumed GA (MFA/OTP and everything was enabled so not sure how it happened but that will need to be investigated after).

Attacker stripped the breakfix account as well and we are kicked out. We logged a job with the MS data governance team whom are barely replying and it's been a day and a half. We've tried calling the number but just get bounced back saying they will look into it. We've asked them to escalate and also asked our CSP to escalate but they said it's with Microsoft. Given the nature of the situation is there any other ways you guys have been able to escalate this to reclaim the tenancy or at least kick out the attackers as fast as possible. We are able to prove ownership of the business etc with domain records/documents etc asap.

Given the no updates I'm straight up thinking of heading to the Microsoft office and sitting there until they can find someone to escalate the case. Anyone had any experience of how to get this moving?

I’m considering using a company called TeamVenti. They provide cloud-to-cloud transfer, copying, migration, and other related services. In my case, I would be copying data from one cloud environment to another.

They’ve asked for Global Administrator permissions on both the source and destination environments to perform the migration.

My question is: Is it normal or standard for a cloud migration company to require Global Administrator access on both sides?

Have there been cases where issues arose from giving a vendor this level of access, or am I being too paranoid?

We have HP elitedesk PC's, and for several months have an issue on most of them where, after a restart, you have to sit there for almost a full min while the ethernet symbol blinks, then goes to the disconnected globe symbol for another few seconds, then finally connects to ethernet before you can enter your login password.

This has cause users to get locked out of their accounts often, because they immediately enter their password before the PC reconnects, and obviously it does not let them in, so they think they mistyped it, and type it again and so on.

We just got new PC's, which are Lenovo ThinkStations, but running into the same issue. I have tried:

going into device manager, unchecking the "allow the PC to turn this device off to save power" under the ethernet adapter

swapped ethernet cable

Running the following powershell script:

New-ItemProperty -Path "HKLM:\System\CurrentControlSet\Control\Power" -Name "PlatformAoAcOverride" -Value 0 -PropertyType DWord -Force

None of those fixed the issue. Doing some more research I found that spanning tree can cause this , which we do use, but does that mean just by using spanning tree we are forced to just accept this long wait to simply login to PC's? Some user's are understanding, but a good number are frustrated cause they have to sit there and stare at the screen for a long time and pay attention to the ethernet symbol before they can login. Surely there would be something in spanning tree that would at least cut this time down from a whole minute right?

We use Microsoft 365 for our email etc.

Our web team have lots of Google accounts they use for AdWords, Analytics, Search Console, TagManager etc.

How do you manage these? Because they've set them up without asking first they've got multiple gmail.com accounts which I don't have access to which is obviously bad if someone leaves the company for any reason.

Got a new job moving from an MSP/small business environment to a global enterprise. Anyone that's done the same, got any tips for me? I'm joining in a EUC role, super excited but a bit nervous at the same time. Been in really generalist my whole career so far for small biz so this is a bit change.. Who's been there, done that, what can you tell me?

We are looking at a handful of options for managing the automation of certificate deployment/updates across our enterprise and OT environments.

I am hoping to have a lab environment set up by the end of the year with at least one reliable ACME tool that can push certificate updates to OT software, servers, workstations, etc...

Primarily use AB and Siemens controllers and HMIs, Ignition, Canary, and Windows IoT, Windows Server (2016, 2022), and Windows 10/11 pro.

Anyone have good recommendations?

Not sure if this is a Veeam-specific thing or just backup monitoring in general, but I'm curious how other admins handle it: our backup tool sends one status email per job, and with enough jobs that adds up to 30-50 individual emails a day. To actually confirm nothing failed, I end up scrolling through all of them manually every morning.

The "proper" monitoring tools that would consolidate this look like they need their own server/infrastructure to set up, which feels like a lot for what's basically "tell me if something broke."

For those of you running backups in-house (not as an MSP) — do you have this solved, or is manual scrolling just the norm? And what happens when the one person who usually checks this is out for a few weeks — does someone else actually cover it, or does it just... not get checked?

Apologies for the wall of text…

Our phone system is a complete disaster. I need help wrangling it all without disrupting users. To understand my problem, I’ll explain the current process, and hopefully you’ll see how flogged everything is.  

All users receive a company cell phone, and a provisioned physical company office phone with its own office phone number. Clients often only know a user’s office phone number. Since 2020, we’ve forward all users’ physical phone lines to the user’s cell phone. This way, clients call a user’s company phone number, and that user can answer on their cell without the client ever knowing our user’s cell phone number. 

When a user leaves the company, we retire their cellphone, and then re-provision their physical phone and re-circulate their office phone number. Cell phones are managed by Verizon. Desk phones are managed by some old school phone company vendor (useless). 

The problem: we have an antiquated internal process where someone (idk who, maybe executive assistants) drags a specific Exchange public folder called “CONTACT LIST” into a new user’s Outlook contact list, so the new user has all company contacts sync’d through Outlook on day 1. 

The actual contact list is COMPLETELY unmanaged. What DOES happen in reality: User A is hired in 2015. At their hire date, they ingest the contact list. In 2018, User B leaves the company. In 2019, User C joins the company and receives User B’s recycled office phone extension. User A, from their cell phone, calls User C to introduce themselves, but their outbound caller ID says “calling User B” because their contact list hasn’t been updated since User B left. SOMEBODY KILL ME. 

This whole process of copy/pasting a public folder contact list is completely untenable. I just don’t know how to fix it for existing users, or how to move forward away from this mess. 

Should I even worry about resolving this for existing users? 

Assuming I scrap this whole process, what’s an appropriate solution to replace it? 

I’m ready to completely change phone vendors, as they don’t do ANYTHING except turn phone provisioning into a 2 hour problem, when a modern solution would be far more efficient. 

What the company needs: users to have everyone’s contact information in their company cell phone at the date of hire. IT needs to be able to recycle phone extensions, and users should see the recycled extension properly updated in their phone contacts to reflect the newest owner of that extension. Somehow…

edit: thanks for advice. My earlier research about redudant power supplies being on different voltages was wrong. I'm going to go with 2 x SRT3000RMXLT-NC with one of them having a 120v transformer.

UPS Setup for New MDF at Our New Facility - Looking for Feedback

Good morning everyone,

A few days ago I posted looking for advice on a UPS setup for the MDF at our new facility. One of the biggest points of feedback was that my original plan only included a single 120V 20A circuit, which understandably raised some concerns.

Fortunately, we're still in the construction phase, so I was able to have the electrical plan updated. The rack will now have two dedicated circuits:

• 208V/240V circuit

• 120V 30A circuit

Rack Equipment

For context, we're a specialized vehicle dealer and service center. We're not heavily IT or office focused, so I don't expect significant growth beyond adding a few phones over time.

Equipment in the rack:

• 4x UniFi Pro 48 PoE switches

• ~26 cameras

• 7 access points

• 20 desk phones

• 2x UniFi UDM Max firewalls (HA pair)

• Dell dual-CPU server with redundant 600W PSUs

• UniFi Enterprise NVR with redundant PSUs (max draw ~450W)

• Fiber modem and cable modem for failover

• Miscellaneous equipment (monitor, sensors, etc.)

UPS Plan

After a lot of research, I decided to go with refurbished APC units from GreenlightUPS, a company that was recommended to me.

Primary UPS

APC SRT3000RMXLT-NC (208V)

• Includes one SRT96RMBP external battery pack

• Will power everything in the rack that supports 208V operation

• This includes the primary power supplies for the server and NVR, along with all four PoE switches

On paper, if every device was drawing its absolute maximum load simultaneously, I'd be pushing the limits of a 3000VA UPS. In reality, my PoE utilization is relatively low, and the server and NVR rarely approach maximum power draw, so I still have a comfortable amount of headroom based on my calculations.

Secondary UPS

APC SRT3000RMXLA-N (120V)

This unit gives me standard 120V outlets and serves as both additional battery capacity and redundancy.

Planned connections:

• Secondary PSU on the Dell server

• Secondary PSU on the NVR

• One UDM Max firewall

• Any other 120V-only equipment in the rack

Long term, I may add the UniFi Redundant Power System to provide redundant power for the switches as well.

For now, if the primary UPS were to fail unexpectedly, I would still have core network and server functionality running, and I could manually move switch power if needed.

I really wanted to buy everything new, but a comparable setup would have easily pushed into the $10,000-$12,000 range.

The refurbished solution comes in at under $4,500, including new batteries. GreenlightUPS claims all units ship with freshly installed batteries that aren't put into service until the unit is sold. They've been in business for roughly 40 years and seem to have a solid reputation from what I've been able to find.

Questions

1. Does this seem like a reasonable approach from a capacity and redundancy standpoint?
2. Is there anything you would do differently if you were trying to stay around the same budget?

Thanks for any feedback.

I've been working in large orgs right out of college. I clawed my way to sysadmin title and responsibilities before an acquisition axed their onsite IT team. My current job is just Support Specialist, with like 14 years of IT experience ranging from help desk monkey to migrating data, managing a small help desk, and being the break/fix tech.

I'm a candidate for a sysadmin 2 position with an MSP and it's 100% remote. I was asked by the IT director why I was making this move. He said it seems backwards since people from MSPs go to larger orgs, according to him. I lied and told him its not about the pay and the job title; because it 100% is. I would be making just $8K more.

I've never worked for a MSP, but I've always been the resource and the point of contact for the help desk MSPs my orgs have had. I've worked healthcare and academia.

Am I walking into a whole different type of circus with new clowns?

Im 26, mainly interested in networking, and ccna, and after trying my chances in diferent companies mainly on helpdesk support, ive finally landed a pretty demanding job.

It seems the scope is almost everything. AD, wild and constant breaking ERPs, Networks and all that is related to it, Databases, hand scanners, even fusion splicing fiberoptic machines. Nonexistent margin of error on mamy things. Revolting printers, many kinds of it acctually. Users complaining about everything, constantly.

People much older than me, with much more expirience, cant operate on their own system, becouse it seems, the system works diferently almost everytime, and constantly figuring out what to do to make it working. Then hearing "you should know this..." and its the thing i see first time of my life.

Programmist, helpdesk, networking guys, electronics, and for some reason HR, arguing with eachother about most efficient way of using this system, deployed only on production alone, reaching pareto points, then tipping over again.

Updates deployed randomly, breaking things, then fixing whats broken again, breaking something else, indefinietly.

Its, demanding, i learn something everyday, and everyday is a simultaniously a disaster, then detective job, then contacting people everyday, to fix things, then enlightment, and solution. And the preassure of time all the time is enourmous.

Is there no celling? No balance? Is it like this everywhere?