u/CeC-P

I think I know the answer to this but...
Let's say someone were to repeat the experiments from 2013 where they physically separated 2 quantum entangled particles with synchronized atomic clocks then caused one to be measured, thus collapsing the state on the other one or whatever it's considered to be happening, then later compared the times - and found that it was truly simultaneous, or on the low end, 10,000x faster than the speed of light.

But this time one of the particles is on a spaceship going 99.499% the speed of light, so causing 10:1 time dilation so 10 seconds pass at nearly non-moving velocities for every 1 second in the ship. The spaceship then is told to come to a complete stop at the exact moment that the quantum particle collapses, this meaning the other one's state was "observed" or whatever.

Then the (relatively) unmoving particle's reading is printed on a piece of paper and blasted off at around 41.66% the speed of light so they only experience a mere 1:10 time dilation, but covering the absolute distance of the first ship from a third party perspective and from their own perspective, faster than the first ship. Would they then arrive at the 2nd ship's location in space before the result happened and be able to give them the slip of paper with the result, thus predicting the future?

I got a call from one of our rural customers at this MSP and their coaxial internet was down again. They lose like $1000+ an hour if they're down so they told me they called Verizon, their business cell provider, and asked if they had something. I assume a salesman mentioned it in the past. After one phone call where I said "our networking guy is out today but I'm pretty sure you can plug just about anything into the WAN port on the Fortigate and it'll work" they called back an hour later saying they got an a XC46BE. I've never even heard of that family of devices so I thought this is gonna be a shit day.

Luckily for my non-networking specialist self, was easier to set up than a 54g linksys in 2002. Basically it just jumps on and spits out wifi and ethernet. I ran a test on my non-verizon smartphone and got 16mbps to the tower. Then I hooked my laptop up to the device and got 220x40 so fuck net neutrality I guess. It even has a battery and their switches had UPSes so that's a bit interesting.

I slapped it into the Fortigate and tada, everyone's back online...with about 10mbps and 500ms ping time. I assumed all their Outlooks were syncing at once or something but they told her the device can do about 20 people. They have about 10 highly active computers. Pretty unimpressive for $350! But they intended to return it when the outage was fixed and their rep said that was fine.

The very millisecond I walked out the door, the ISP truck showed up and started messing with the box on the front lawn. Awesome use of my time.

But I keep hearing about these magical devices that can switch over to 5G and we do have WAN1 and WAN2 on the Fortigate so they're considering keeping it and programming in a switchover of some sort. I assume they do that. Anyone have one that doesn't suck? Because this one impressed me until it was actually in use. I saw one at a trade show years ago that was a UPS + 5G modem. That sounded kinda neat but so did this Verizon device until it performed poorly.

Got in a ticket that started with "new person can't import a catalog into the database and nobody knows why" to Support says it's a known, unsolvable problem in the 2023 software and we need to upgrade the entire company to 2026 and migrate all the data and they need it RIGHT NOW.

Ticket gets cancelled. New person went NSNC as they say (no show no call). Yeah, don't hire 20 year olds. I could have told them that. So now I have weeks to fix this one!

I realized I can think of at least 3 other nightmare tickets from hell that turned into a train wreck but got cancelled because the person quit or got fired.

Also I accidentally scratched the company car a few weeks ago and did a bad job repairing it with a paint pen. Then someone hit a deer in it and totaled it. I'm batting 1000 this month.

I suck at network and my knowledge is intermediate at best but I can't solve this one.
Customer at our MSP has a fortinet firewall identical to ours that we use here at the MSP office, same firmware version, etc.
They call up and say "We can't access prodemand.com" which is an automotive parts and labor quoting database site that TONS of dealerships use.
I load it here just fine, SSL cert is GeoTrust, good till Sept 14 2026, domain matches, etc. No web filter flags.
On their network, instant "cannot load page" error. I try a dozen other sites, SSL working fine, no fortinet intermediary cert listed, etc. It's just that one website.
Security log on the Fortinet shows tons of blocks, saying "SSL connection is blocked due to unable to retrieve server's certificate"

Mountains of troubleshooting later, I make a firewall rule for internal to WAN (and put it above the normal internal to WAN rule) with an address group of the site, the login domain, and the database's UI's subdomain. The rule simply says don't inspect SSL at all.

Boom it works instantly. Then they called back because WIFI wasn't included in "internal" lol oops. So added that, boom, laptops can load the site too now.

I ran through some basic troubleshooting and traceroutes and stuff and nothing stood out as problematic. I verified no man in the middle attack, as it sees the same cert I do here.

And AI thinks it's an ISP issue but AI is dumb as hell and for the record, rebooting the firewall and the modem didn't resolve it so I'm skeptical.

But zero other websites are having this problem and we don't see the problem from our office, using the exact same firewall with same firmware version. How is this possible? I'd really prefer to get rid of that rule because it's a crap workaround and we had to also turn antivirus and other filters off, since it requires SSL inspection.

I noticed in the last 20 years or so that there's 2 types of IT hardware support workers:

The precision "as-is" hardware ninjas
and
The "who cares" dongle goblins

My earliest 3 IT jobs were just swapping out old computers for new ones on a contracted team. It was 90% of our job. I would replace the users' reference sticky notes and desk decor so perfectly that we got at least a dozen complaints that we forgot to replace their PC and monitors.

About half of the rest of my team simply could not concentrate or didn't care. And this was around 2009, before Tik Tok and before most people gave a crap about smartphones. It was just their personality. I mean, I can't remember names so maybe it's just how we're wired.

The #1 problem was forgetting to transfer the wireless mouse and keyboard dongle(s) to the new computer. We got back at least 50 that still had them in the USB ports and it was always the same 2 people. We nicknamed Donald: "Donny the Dongle Goblin" because he collected them like they were shiny treasures to be horded.

Fast forward to my last last position - We had headset disconnections, wireless mouse interference, AP interference, and we traced it to the morons that rolled out 25 sets of identical wireless mice and keyboard on top of 4 APs and 100% wireless headsets, all using 2.4GHz. Plus people's smartphones on the guest wifi. Every time a flat-sided box truck or semi rolled past, it reflected the neighbors wifi in a way that flooded the spectrum and disconnected phone calls. For the entire 3 years I was there, they refused to stop ordering wireless mice and keyboard. Double digit percentages of our budget were replacing non-programmable logitech sets with missing or wrong dongles at like $40+ each. I don't know how that's even possible. It's laptop + dock on a desk and nobody traveled with them. Where did they go? How did they get mixed up? I HATE DONGLES! Can we just direct wire the damn peripherals so Donny and stop hording them and we can stop spending money replacing perfectly good keyboard and mice sets?

Little behind the scenes for all you IT newbies on this wonderful Friday. This is likely why your IT budget makes absolutely no sense and has nothing to do with the company's financial status, although the RAM shortage affects this heavily. But the story is from 2024. My last company was making record profits, over $20M more than the previous year (so about 40%) with $100M in the sales funnel for sold jobs.

But the IT dept was approaching over-budget because our useless dumbass CIO filled out the budget wrong and forgot about a $9000 license renewal. So everything was on a spending freeze in IT solely and exclusively to make the numbers look good. After absolutely going off on everyone about how this makes no sense, is losing us productivity, delaying jobs, costing us income, causing outages, etc and we're single person owned and not publicly traded so who the fuck are making the numbers look pretty for, someone finally leaked me the truth.

This was solely because the rich, retiree asshole board members got performance bonuses based on benchmarks and this was one of them. So if our dept went over budget, they lost part of their bonuses. They cared more about their income than the company's health. This is the owner, the former CEO, some of their friends, and some external 3rd party entities that I knew nothing about.

I have an idea - revise the fucking budget at the Q1 mark. Budgets are a guess. You don't "go over budget" if shit got more expensive so you adjust your guess to spend more on the shit that got more expensive! That's just business. Adjust prices and margins accordingly. This was not a 10,000 person bloated company spread all over. It was about 290 people. So they added some emergency special budget special condition whatever spending, outside the budget, make the numbers look pretty thing and dropped some serious cash on all those laptops.

Those laptops were $810 a piece btw in 2024 and are now like $1500+. So you're welcome, asshole who fired me 2 weeks after I asked for a raise then said this is unacceptable and I'll be looking for a new position until they reconsider because me rent went up 18% two years in a row.

Btw them firing me without a replacement when I did all the server and VM host maintenance, security audits, hires and fires, backup management, all level 3 tickets, some networking, and basically maintained ALL systems (for $24/hr), the remaining staff member told me it ended up costing them around $800,000 in damages and losses and outages and delayed jobs when everything I maintained the the other staff refuses to cross train on all broke. Probably should have given me a raise and replaced the CIO since he was shit at his job (but was there for 30 years).

Cheap, illogical, shitty companies run by greedy self-serving assholes are a trap and you need to get out as soon as possible because you'll run into stuff like this. And it goes 10x if you're publicly traded because then you REALLY need the numbers to look pretty or else.

But it's mostly my fault. TL;DR even if you're a 15 year veteran of IT support, ask the user WTF they're talking about and ask for a screenshot or error.

Someone puts in a ticket, says they can't open an excel doc. It was someone from the same department and same day as a read-only expired Sharepoint issue so I assumed it was there and pointed them to the directory where we moved everything from that Sharepoint/Onedrive.

Nope, unrelated. It's on the file server under accounting. So after getting the file path, I notice she's not in a permissions group that has access. So I grant her access and instruct to log out and back in to get the permission token.

Nope, that wasn't the problem. Still reporting as "locked." So I thought oh, it's the "another user is editing this" lock thing because that happens sometimes there on network drives. So I clone it, open in Libre, convert it to ODS then back to XLSX, and delete the original.

Nope, still reporting that it's locked. This user is impossible to contact by any method invented by mankind. Only way is to set one of those cartoon cargo net traps outside their office and snag them in it. So I request a screenshot via email, the only communication method they use.

She finally sends back a shot of "this one specific third sheet inside the file is locked/write-protected. You need a password to unlock it."

I use one of two known tricks to unlock that one sheet without the password. It works. Finally closing the 1.5 week old ticket.

Btw the sheet has in cell D1 in huge letters with colored highlight "This is the password to edit this sheet:" followed by the password in the next cell.

I guess that missed that.

The Ninja RMM update last night seems to have somehow translated into the Ninja remote agent breaking on just specifically domain controllers. It might be server OS specific. So now we can't get into any of our MSP clients' servers to do new hires. And it's Friday. Great.

Also, it's reporting offline status (probably the real problem) so we can't even attempt to use Splashtop or RDP. We have to RDP into the server from a client computer onsite, which hopefully is enabled at every customer. Thanks, Ninja! I hate you. Wish we used Connectwise.

All of a sudden this windows 10 computer (yeah, yeah) that just needs to VPN into an offsite server to run a time clock app started failing to do so. It connects but after you connect, you can technically log straight into the firewall by IP, so I know the connection worked, but I can't get any DNS to load, no websites, and can't ping 8.8.8.8 for example. As far as we know, nothing changed.

So I exported the profile, installed Forticlient 7.4 on a brand new Windows 11 25h2 virtual machine at our office, which is a different IP and ISP, and it connected fine but also killed all ability to load websites, etc. We don't think anything changed on the firewall and it reports healthy so not sure what could cause this all of a sudden. And automatic firmware updates are actually disabled on the firewall (so I'm patching it to 7.10 then 7.13 sequentially tonight off-hours).

Anyone see this weirdly specific issue?

https://preview.redd.it/xkvxbqi55gah1.png?width=728&format=png&auto=webp&s=3a12b54121125d4bebcc8a1c846dcd15355801ea

I assume they were trying to hit the "this will explode in pyrotechnic hell-blast if it's damaged" square directly but missed a little to the right.
Btw my cousin worked for a delivery company and said 1000% the a-holes loading the trucks are behind this.

I work for an MSP and we just deployed about 5 new printers this year. The customer has now destroyed 2 of them with Amazon counterfeit cartridges and one is refusing to "connect" to the chips and refusing to print on 3 of 4. So now they're out the money for the counterfeit garbage, the money for 2 printers that the carts exploded in, and we have to drive out and attempt to repair/clean one after they get real cartridges for it and hope the system that puts the toner onto the drum isn't damaged by badly out of spec molecules, which is what it sounds like.

Good thing they saved so much money shopping on Amazon for "just as good" cartridges. American medical care provider btw. That should scare you.

There is no right answer here but I'm wondering what the strategy is for you guys. I've had CIOs say they want all the same model laptop/desktop so they can stop tracking down individual problems with specific hardware. Obviously that's incorrect and I can assure you that it'd have been easier to flash 20 USB C power delivery chip firmwares instead of 100 when users are in the field. And then the same again with defective Ryzen drivers that kept crashing our remote control. And we have to fix it on remote control. Also, I can track down specific hardware defects after never seeing the same laptop twice at my non-commercial computer repair store for 8 years. But I believe my skill set is unique in the corporate IT world.

Obviously imaging efficiency is a factor. I just don't like putting all my eggs in one basket, however, if I'm charge of purchasing then the best laptop for the price and performance is very obvious. Going one step down is unacceptable and one step up is a waste of money on the order of $100+ typically. I am VERY sick of diagnosing specific motherboards' random incompatibilities with DP passthrough on random models and specific firmwares of docks though. If we had all the same model, we'd have simply fixed it, everything is flashed to a version that works, and there's a lot less tickets.

I think right now you just buy either nothing or what's in stock for the cheapest because Sam Altman ruined the entire world so perhaps describe your strategy before that, and if it backfired spectacularly or worked great.

Just kidding. MSP here and it turns out it was actually the fact that the two main switches are under the secretary's desk, because duh, where else would you put them? And she runs a space heater if it's below 85F in there. Turns out snagless CAT6 housing is also known as heat shrink tubing and it will squeeze the plug and eject the Ethernet cable on its own, if hot enough for long enough.

Yes, we have told her it's not ideal to do that. No, she doesn't care.

I picked the wrong week to stop sniffing glue.

This is all the data I could compile on the problem. I had to do heavy research on it because I never looked into or cared how any of this key and cert and boot time stuff worked until it affected my job directly. It was just monopoly abuse against Linux in my opinion. I'm no expert on this so if I get anything wrong, leave a comment, and I'll edit this post.

What happens if you do nothing

June 24th - absolutely nothing happens. Any system running a modern Windows OS will work because it was loaded with the old Secure Boot allow list and the boot loader is on it, obviously, or it wouldn't have been able to boot.

Exceptions: none that I could find

Long term if you do nothing problem #1
You try to install Windows 11 26h2 onto a computer that never had a BIOS firmware update with the list of new allowed certs/private keys/whatever. It fails to boot from the flash drive installing it, for example, because the new boot loader isn't on the allow list.

Solutions:
- turn off Secure Boot and install it anyway. Windows currently does not require it in order to install. Very little software needs secure boot to be on in order to run. The only ones I'm aware of are anti-cheat systems from EA, Activision, and Blizzard. OH NO. But the following may throw errors and/or not work in Windows:
Memory Integrity (HVCI)
Device Health Attestation tests
Windows Defender System Guard
Secured-core PC features
Enterprise compliance policies in general

Install the OS then turn on Secure Boot after the OS is installed.

Note: If you turn on Bitlocker then flash the BIOS from within Windows then turn on secure boot in that exact order, it will ask for the recovery key at next boot. Leave Bitlocker off until the end of the process if this is your plan.

- boot to a windows PE like Bart PE and then patch the bios with whatever simple standalone patcher thing Dell/HP/Lenovo/whatever provided. Then install the OS after that.

- Don't load an OS and first flash the BIOS from within the BIOS using a usb drive containing the new ROM. This is more commonly a capability of non-OEM MSI, ASUS, gigabyte motherboards, not corporate OEM hardware.

Long term if you do nothing problem #2
MS finds a vulnerability in the old boot loader/cert system/encryption method and adds it to the revoke list. Maybe quantum computers crack it or something. They push out the revoke list via windows update. Now any system without BOTH the new BIOS firmware and new patches on the Windows OS side instantly overnight won't boot until you turn Secure Boot off in the BIOS config, causing a near Crowdstrike level event.

IN MY OPINION, this is not a realistic scenario.
- how did a system that's stuck on 23h2 for example receive a windows update at all after end of life?
- why did the windows update that revoked the old boot loader run but not the much earlier one patching it run?
- okay, a computer has windows updates turned on but filtered by a 3rd party patch management system at your company, it can't patch to a later build of windows because there's not enough space on the SSD, but all other updates are working. You don't run Dell Command (for example) to patch the BIOS because BIOS patches are scary and then you blocked the same firmware patch pushed out from the OEM via windows update in the critical drivers channel. Then you accidentally let through the theoretical future insecure boot loader revocation update. In that case, you're a terrible sysadmin. You get to now get off your ass and go turn off Secure Boot one a time, per affected computer so they can boot and then clean up the problem later.

But realistically, a tiny, specific secure flaw that still probably needs a UAC bypass to run anyway is found so Microsoft's solution is to release a patch for OS versions that isn't supported/getting updates anymore and the sole and entire purpose of the patch is to make the OS not bootable because the OSes own, current boot loader is now on the revoke list. The next day the media says Crowdstrike 2.0, everything is down, it's Y2K, OMG, except MS knowingly did it on purpose. Then accusations fly about it being planned obsolescence and the feds throw the book at them. Why are they even patching a system after saying "we're no longer providing patches for this build of Windows." I don't think the terms and conditions you signed, or arbitration limitations would prevent MS from getting sued out of existence if they purposely, knowingly released an update that destroyed older computers "for security."

Realistic side effects:
All your old bootable utilities like PEs or Macrium (for backup and image deployment) stop working.

Solution: get new ones or turn off Secure Boot as needed (which breaks bitlocker). Yay, more budget expenditure for Macrium 2027 edition or more IT staff time. Oh well. It's still cheaper than RAM.

SCCM and/or Autopilot breaks
It already works like crap on any given day, especially together. Add another problem to the pile. Okay, okay, I know next to nothing about those systems and how they work or how they relate to Secure Boot. One company I worked at used both, they work like crap, and MS keeps changing them. Oh well, good luck. You should have pushed out BIOS updates before June 24th then and maybe reconsider using magical MS tools that do everything perfectly and make your life easier because they never do.

Solution: (my best guess) Patch the BIOS manually via a working, modern PE. It takes about 3 minutes if you're in front of it. If your users are remote, then that sucks. Or use the backup method for imaging and deploying new computers that you already have in place for when either SCCM or Autopilot fails. If you don't already have one, you do not live in the real world.

Outlyers:

1. Extremely custom boxed hardware/software solutions running windows with a very custom motherboard with kiosked OS environments that aren't supported by the manufacturer anymore, so no BIOS updates to get the new allow/revoke list for secure boot loaders. People are throwing a fit over these since you typically can't turn off Secure Boot in their custom BIOS configs or you don't even have the BIOS password in the first place.
   - well they either are or they aren't getting Windows updates if they're highly controlled, custom solutions made by 1 vendor. So how would they not get the new Windows updates that fix the problem but do get the Windows update that revokes boot loaders? Block patching for them completely (many methods for this) until you can replace the systems or just live with the horribly insecure, locked down, canned garbage that you shouldn't have bought in the first place.

2. You had a laptop brand new in the box made in 2023 and you deploy it in 2027 and image deployment fails because the installer uses a newer boot loader.

Solution: WTF is wrong with your company? Ever heard of depreciation and CMOS battery failure? Then update the BIOS manually before deploying the new OS via the methods outlined above. If fact, you don't even need a PE. Just boot to the OS it came with, patch the BIOS, then pave over it with the image deployment method you already use.

3. Anything listed already except the users, the computers, etc are 100% remote and management thought it would just sort of work itself out somehow magically and nothing major would ever happen.
   Solution: Yeah, that is a problem. You probably should have planned for that, as 100% remote IT management inherently doesn't work. Oh well, hope you have a good rate with a shipping carrier. Management knew what they were getting into with that business model.

4. Something goes wrong with the patching chain and it somehow causes Bitlocker to trip in large numbers.
   Solution: that already happened.

5. Your company uses custom Secure Boot keys
   Solution: why the **** did you think that was a good idea? Time to find a new job!

6. The hardware is 10+ years old and the makers never made a BIOS patch to work with the new certs/boot loaders.
   Solution: Yeah, it wasn't going to run forever. Either a failed hardware component or other security vulnerability was going to take it down eventually. It's probably susceptible to rowhammer/specter/meltdown/etc right now anyway. You should have been looking into replacing it, despite difficulty and cost, already. Now you're being forced to do so. Or block that Windows update on it or reinstall build 25h2 or go turn off Secure Boot in the BIOS by hand, in person, before June 24th.

Hopefully I didn't miss any fringe stuff or important details but let me know and I'll edit the post. 0% of this was written by AI and the whole post is public domain licensed.

Setting up 5 brand new Yealink T54W phones with Ring Central. I provisioned them so they're all in the admin portal on the web. They updated the firmware as part of that. Then I noticed one of those two things reset the admin password to default. Now it's also complaining that user "user" and user "var" are using default passwords.

I assume user will be filled in when I assign a user/extension to the phone. But should I change var, as that's an internal type of user. Also, should I then change the admin user account on the phone via the web interface of the phone itself on our local network, or will that somehow lock Ring Central out?

Also, can I change any of these centrally from Ring Central's admin page? It'd be a lot faster if I could.

I just found out they technically do exist and some of them are even not scams/counterfeits. I was tracking some down for use in a small battery pack that will have VERY high cycles. Then a source told me that 18650 LiFePO4 cells tend to be rated for 3.2V and top off at 3.65V.

That means they won't work in a flashlight or traditional battery back with USB charge. But that's fine, I was going to put them in a small 12V arrangement. But then they'd have to be 4SxP instead of 3SxP arrangement to get to over 12V. That's kinda weird. I have to check the input range on my intended inverter because 14.4 is pushing it for some. Anyone else use these ever and anything else I should watch out for?

Remember the late 1990's when people would steal 128MB sticks of pre-DDR RAM worth about $300 each from computers before resigning or getting fired so they put padlock loops on the desktop cases? Yeah, they're like $400 a stick now for 64GB setups. We had a request to do so by one of our MSP customers after we can't really prove it but we're 99% sure someone stole a stick.

Considering I can get past a dollar store bulk padlock that small with a paperclip, I instead put in an RMM rule that says send a high priority alert email if the RAM on a system falls below what it is now by more than 10%. I had to hard code it since that wasn't a trigger template for some reason.

Anyone else already run into this and doing something similar? For everyone else, not a bad idea.