r/talesfromtechsupport

I do basic IT support and got a call from someone saying the office printer had been “shrinking everything” for two weeks.

Not one document. Not one app. Everything. Emails, PDFs, forms, random screenshots, the whole lot. She said it like the printer had developed a personality problem and was now deliberately making documents tiny out of spite.

So I remote in and ask her to show me what happens. She opens a PDF, hits print, and the preview is basically a normal page surrounded by enough white space to rent out as a studio flat.

I check the print settings. Scale: 25%.

I ask if she changed that recently and she immediately says no. Very firm no. The kind of no where you know this machine has already been blamed in several conversations.

I change it back to 100%. Print preview looks normal. She goes quiet for a second and then says “oh, so it fixed itself?”

No. It did not fix itself. I fixed the tiny little number that had been telling it to make everything tiny.

Then she says she thought zooming out on the document would “save paper” when printing, because if it looked smaller on screen, obviously it would print smaller too. Which, to be fair, is almost logic. Bad logic, but logic.

The best part is she still asked if I could “keep an eye on the printer in case it starts doing it again.”

So now I’m apparently monitoring a printer for signs of relapse after it spent two weeks obeying instructions perfectly

I am a fiber optic engineer. And this datacenter will be the death of me. Recognizing the big words will bore the crap out of most people, I'll give the abridged version.

This particular datacenter does not train or equip it's employees properly, and as a result, anytime they have a slightly complex problem, they make it worse by trying to fix it.

Then they call me for help, despite the fact I don't even work for them. I am their OSP vendor, meaning I fix problems outside and between buildings. Inside their own building is supposed to be their own problem.

I receive an email. This infamous datacenter tells me they're experiencing an "ORL" issue. That's optical return loss. It means the connection is too shiny and too much light is returning the way it came, backwards, instead of going forwards through the fiber optic cable to the destination.

I tell them it means their connectors are dirty and to clean them with proper cleaning supplies. Fun fact: they do not have proper cleaning supplies.

Days later they follow up, telling me the issue is now a 10dB degrade. That gives me pause. That could actually be an issue between the buildings I would be responsible for. A degrade means the light going from one building to the other is too dim when it arrives, some of it has been lost. 10dB is not a small amount, it means 90% of the light is missing and only 10% is getting to the destination.

I show up. I start my troubleshooting by asking a lot of questions. The answers I get confused me. The equipment readings I get confused me. Finally I realize what's happened.

On a previous trip I told them they had bad patch cords, they would fall out of their plugs if you so much as breathed at them.

Following my advice, they replaced them with proper cables. So far so good.

But one of the circuits didn't come back up fully. They properly diagnosed the new cable was dirty and needed cleaning.

Mistake number 1- I told them the cleaning supplies they had on hand were terrible, and likely to make things worse. Standard cleaning solution is isopropyl alcohol. I was told that was a hazard and they needed to use an alternative cleaner. This alternative cleaner is a mixture of Propyl Acetate, an industrial solvent that is itself flammable and emits hazardous vapours that are also flammable. Ethanol, which is literally fuel. And- Isopropyl alcohol!

This cleaning agent leaves behind some oily residue that- causes ORL issues because it's shiny. So they cleaned this fiber over and over and failed every time, and concluded the problem must be elsewhere- completely ignoring that I had already told them their cleaning supplies sucked and were incredibly inadequate.

Now if their cleaning supplies weren't awful, that might be a reasonable conclusion. So they performed a loopback test.

The connection normally goes equipment, patch cord, rack, cable leaving building, rack in next building, patch cord, equipment. A loopback test is purposefully looping the patch cord back to the transmitting equipment by connecting it to itself at the rack.

When you do this, you add an attenuator. This is a special plug that adds loss. The system is set up assuming you'll lose so much light going from building to building, so that short range connection needs to be 'dimmed' a bit. That's what the attenuator is for.

Mistake number 2: they connected the attenuator, and forgot where they put it.

The 10dB degrade I was sent to repair was caused by a 10dB attenuator they installed, and couldn't remove, because they forgot where they put it. And on top of that- they still needed me to clean their damn panel for them.

The remote support team knew all of this, and through tactical lies of omission, made me think I might actually need to fix something that was my responsibility. Instead they used my competence to fix shit their internal team doesn't have the training or equipment to deal with: which are basic essential functions of any datacenter team.

These lazy bastards tricked me into troubleshooting diagnosing and repairing their own shit because they can't be bothered to train and equip their own employees to do their jobs.

I put my foot down and told them this is our last courtesy dispatch. For future calls involving this datacenter, we are working strictly to the terms of our contract.

So here we are yet again. I traded my job in the hosting sector for something slightly less… soul sucking in public sector IT. And as many organizations do these days, we rely heavily on Microsoft for a lot of things.

And that sets the stage for today’s tale of horror.

It started simple enough. A few calls from remote workers unable to authenticate into Citrix using Microsoft Authenticator. Annoying, sure, but not exactly catastrophic. We’ve been having enough issues with Microsoft lately that we actually have a backup MFA method for this exact scenario.

So the affected users log in through the backup, we log the tickets, escalate them to the remote work team, and move on with our day. Slack starts filling with the usual Microsoft memes. Business as usual.

Or so we thought.

Suddenly systems start dropping like flies.

First Teams starts screaming that we have no network connection. Which is impressive, considering we work entirely through Citrix. If the internet was actually dead, our entire remote desktop environment would be gone too. So clearly something else was happening.

Then the first colleagues finish calls. Me included.
End call. Move cursor to call system. Click “Done”.

Nothing.

Click again.

Still nothing.

Okay… That’s not great.

Now we can’t change status anymore. People start getting stuck in break states, active call states, all kinds of nonsense. Then internal sites relying on Microsoft authentication begin failing one after another. Shortly after that, external sites stop loading entirely.

At this point everyone starts asking each other:
“Wait… are you guys seeing this too?”
Turns out nobody was hallucinating.

Meanwhile management starts mobilizing. The call queue climbs from 50… to 75… to 100.
Which sucks, but hey, we have a fallback phone system…

And you guessed it, that can only be enabled by going to an external site.

So for the next 45 ish minutes we mostly sit there watching the infrastructure equivalent of a medieval city burning down around us.

Eventually, after a long call with telecom, the fallback system finally comes online.
And at that exact moment, our primary phone system decides it’s healthy again.

Of course it does.

And here I am now, two hours later. Still logging tickets. Still telling remote workers:

“Yes, we know Microsoft Authenticator is still broken.”

And so Monday passes by.

Now I sit here on the train ride home preparing tonight’s D&D session to recover mentally from the experience, writing this post as a form of therapy.

So as I’m already in a medieval mood, I leave you with the wisdom this day has granted me:

Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.

This tale involves probably the worst raid lead I ever had in ffxiv and his know it all friend who kept countermanding the advice I was giving him. I wont get into the reason he was the worst raid lead I ever had. Just know he was a bad tank, and a bad raid lead.

The tech issue was complex for him. Whever he plays ffxiv in discord, his game freezes up. He has no issues if he is not in discord. He uses a set of senheiser (sp?) headset and a dac6. I asked him to bypass the dac6 one day as a test.

He said he did this, but did not. Issue occurred. His best friend, and girl he just happened to be sleeping with, both were telling him I was an idiot and had no clue what I was talking about. The issue HAD to be video and not audio.

Even though the issue ONLY occurs when he is in discord and playing ffxiv.

Second solution. Turn off hardware acceleration.

He did not do this. Ill give you one reason why.

Third freeze happened mid raid. He asked me what he could do. I gave him team viewer and checked event viewer once connected.

I used a 3rd party website to read the vent viewer log, as I was too lazy to look it up for a raid member, and it showed a flurry of audio errors and permissions errors right at the freeze.

I check discord. Hardware acceleration is on. I turned it off. I turned off all audio enhancements in his headphone settings. I set the dac6 control panel to use 24b 48khz, windows sound settings for the dac6 to use 24b 48khz as he had set it to 32b 48khz.

FFXIV can cause a freeze if there is a mismatch.

I turned off exclusive control for the dac6 in legacy control panel sound options and then had him restart.

Rest of night no freezes. Rest of raid week. No freezes. Two whole weeks of no freezing.

Then his friend got ahold of his system. Ill give you one guess what happened next raid night.

I asked him if discord hardware acceleration is on, or if he turned on the audio enhancements in his headphone software. Yes to both.

I told him to turn it off and turn off the audio enhancements while using ffxiv. You can turn on your custom mixer when listening to your music after, but turn it off for ffxiv. Its causing issues.

Two weeks later.

His friend, the know it all, got involved again and told him that the changes I made obviously did not work as his game crashed on him.

His game actually crashed from using mods and Lightless Sync updating in the middle of limsa with 40+ people near him he was synced to... But thats just my opinion.

I joined discord and heard his friend saying I had no clue what I was talking about and that he should just kick me and replace me with his astro friend.

The static was doing Tea and just gotten past BJCC.

I hear his friend say this and basically tell him dont bother. I will leave on my own. I send a DM to the raid lead telling him his friend is an idiot and if he wants the issue fixed, to leave the changes I made alone.

His response was to send me a HR approved team removal message a day later. Said the group came together and made a decision... yadda yadda. Then a very snarky message that I should just give up on TEA.

I was going to leave it alone. But I am petty. That message to just give up on TEA?

So I cleared the fight 3 days later and just HAPPENED to be near the exact spot he likes to stand in Limsa on his server with my shiny new legend title.

The funniest part. After he saw me in game and said in his discord. "Lightning got his Tea Clear. Yeah he is standing right by me in limsa." People in discord called me out. "He totally did that on purpose to rub it in our face that he cleared."

Before he could answer... He froze up live on twitch again. His friend, the one who constantly counteracted everything I did, suggested he turn off hardware acceleration in discord and turn off audio effects in his headset control panel software.

EDIT: I should not type these up when I am very tired. It was only a few days after I cleared when I saw him go live in twitch. Thats why I went to his location in game. I was being petty and rubbing it in cause he told me I should "Just give up" on Tea.

I work internal support for a company where Excel is basically treated like a second operating system.

Got a ticket from a user saying their monitor had “shrunk Excel” and they needed a replacement. Not that Excel was blurry. Not that the monitor was damaged. The monitor had apparently decided to make Excel too small.

I remote in and ask them to show me the issue.

They open a spreadsheet and, yes, everything is tiny. Like ant accounting tiny. Columns, rows, text, all of it. I ask if other programs look normal. They open Outlook. Normal. Browser. Normal. Desktop icons. Normal.

Only Excel has offended them.

I look at the bottom right corner.

Zoom: 10%.

I ask if they remember changing the zoom. They say no, and then immediately say “I only moved that little slider because I was trying to see more columns.”

Right.

I slide it back to 100%. Spreadsheet returns to human size.

User goes quiet for a second, then says “so the monitor is fine?”

Yes. The monitor has bravely survived this incident.

They thanked me, but still asked if they could get a bigger monitor anyway because “Excel has too many boxes.”

Ticket closed as user education, which is the polite way of saying Excel was not the problem today.

We have a slightly above average userbase as employees in regard to tech skill. Though Ive had to talk some through how to shut down their computers most are pretty good, especially if we provide a PDF tutorial or something.

However some are very good at their 1 app and have no concept of the totality of what theyre trying.

Recent we upgrading our user facing report writer. Simple tool, grab column/row object, drag to report pane, poof data. One of our better reporting users decided they would use this tool meant for basic reports to make a bigger one so they wouldnt bug us. Sounds helpful but she ran into issues.

This was going to be the first report made after an update. So naturally there were some server growing pains me, not a DBA just a server pleb, had to resolve. Figured out those in a couple days. Close ticket. Couple days go by and the ticket is back.

Hmm weird thought I closed that, wait, crap, she reopened it. Oh she just cant export the report. Probably another server issue. Spend probably 20 hours over 3 days looking into it before I ask what she is trying to pull.

She was trying to pull every data point in the server except for customer name, address, etc. Literally payment history, balance due, closed out accounts, days, times, memos on accounts, etc everything on an account except specific identifiable info she was trying to pull.

During all of this the DB and other systems kept going down randomly and we kept having to break from this to look at that. Outage bigger issue than no new things, obviously. Then we learn what she is trying to export and when we line up when she was attempting to the outages theyre in sync exactly.

She didnt understand why the system wouldnt let her do this but eventually gave us the criteria she needed and our Jr DBA had the report done pulled straight from SQL in like 25 minutes.

TLDR; you pay DBAs let them make the complicated stuff and never try to export "all of it"

Happened this week. I’m a M365 SysAdmin. A ticket was escalated to me where the user wanted to modify a calendar’s permissions. Ok. Not a big deal. But I could not locate the calendar.

So doing what any good IT professional would do, I asked for a call over Teams and then requested the user share their screen. Once connected and seeing what they see, I made the request. “Please show me how you access this calendar.”I’m expecting Outlook > Calendar > specific calendar. But nope!

The user opened File Explorer, navigated to the File Server, and through several folders. The last folder led to a file called 2026 Calendar. And it’s an Excel file! No wonder I couldn’t locate the calendar anywhere in Exchange!

After that I suggested that maybe we should consider a real calendar for future use. 🤦‍♂️

I had one today.

We have a product that can be installed locally, or can be used in a cloud environment. The cloud system gets updates often, the local system does not.

There's a way of migrating customers from a local system into a cloud system. It's pretty easy.

This one customer is demanding. Had to try migration to the cloud a few times for various reasons - once a software issue, the other 3 user issues.

So attempt 5 today. I am 3rd level support. 1st level skipped due to customer "value".

Migration worked fine. 2nd level support teams' me shortly after in a panic. A key user can't login. This is prompting the customer to demand he rolls back to local version.

So we persevere, reset the password, try again, it doesn't work on his side. My side - works perfectly. 2nd level support and customer is confused. Other people call login fine, so it's just this one person.

We have a self service website and app. 2nd level support AND customers BOTH say website works but app doesn't. Tried it - app for me works fine. WTF.

Customer still pushing support to roll back. Aggressively.

I ask 2nd level support to tell me what email address they're using to login to the app.

It's different to what was provided. ALL 3 PEOPLE - 2nd level support, the customer's manager demanding the roll back and the key user ALL ENTERING THE WRONG EMAIL ADDRESS.

They use the right email address, and like magic - everything works fine.

They almost rolled back a huge system update because of an incorrect email address.

🙈🙉

These are my two most used Teams emojis at work these days. We have a running joke - our 'monitoring unification plan' would solve so many of our day-to-day problems, because we have such a half-baked system in place that we only get 'some' visibility of our estate (5 sites daisy-chained together with private fibre, 2 of which don't have their own internet connection). But because we have 'some' visibility and not 'none', there's no political will to prioritise the project and it's been sitting in our backlog for longer than I've been here. We do have (near-)real-time service monitoring, metrics collection and log aggregation, but only one person seems to know how to query Kibana, metrics are only used when there's other problems and the real-time monitoring often monitors something completely worthless. One of our low-end single-purpose NASes keeps throwing alerts on running out of memory; my boss' response was, 'can we do something about this annoying alert?' My suggestions were, 'Disable the alert? Disable the polling? Fix the root cause?' Followed by a GIF of Homer covering up his car's Check Engine light with tape. See no evil, hear no evil.

Today is a story of useless monitoring, and it really takes the cake.

Pretty much as soon as our department-wide Teams morning standup began, something went wrong. I say something because we still have NFI what happened, but websites started timing out. So as more details trickle in and we complete standup, a bunch of us remained on the call diagnosing the problem. It turns out that sites are timing out over the VPN as well, which is a good indicator - our VPN is a split tunnel but forces site DNS on everyone. Sure enough, a bit more poking and we discover the old sysadmin haiku remains ever accurate. It was DNS. And this is where it gets weird.

Imagine our setup of 5 sites as a line - 1-2-3-4-5. Site 2 is our primary, 3 is our DR. Sites 1, 2 and 3 have their own internet connections. Sites 4 and 5 don't and have none of their own infrastructure as they're small sites, so they both go through site 3. It's all AD and MS DNS. All 3 sites use the same upstream DNS provider. All 3 use the same ISP.

Well, both DNS servers in site 2 start timing out for every external query. Internal is fine. But the internet connection is up, and DNS at the other two sites are working properly. Wut. So thankfully, sites 1, 3-5 are unaffected, but as the VPN specifies DNS from both sites 2 and 3, browsing via the VPN is affected as well as all onsite in site 2.

So we prod and poke at it for a couple of hours. I'm a Linux guy so I'm not involved in the AD stuff, that's my colleague's remit. We eventually come to the conclusion that the upstream DNS provider is the culprit, but only for this one site. If we switch upstream to Google or Cloudflare, it works properly. The provider reports no known issues and manually querying them from my laptop works 100%. Yet any upstream DNS from the site 2 servers gets no response. Firewalls are ruled out, WAN links are ruled out... Yeah, we have NFI what happened. So eventually we have to just leave the alternate upstream DNS in place.

Now, to tie this back to the opening paragraph, monitoring is a hot topic in my team. Too many of the old-hands have lost the will to do anything about it, but me and my fellow Linux admin are eternally frustrated by the lack of monitoring. So the first thing I suggest is - at home, I have Uptime Kuma querying my homelab DNS servers and sending me a Gotify ping if something stops working. Let's do something similar. It may not give us much additional visibility but it'll at least save the 15 minutes of 'does this work for you?' that inevitably results from diagnosing something on a video call. At the very least it'll point us straight at the faulty DNS server(s).

I open up the real-time monitoring for the site 2 primary DNS server (also domain controller, of course). And I immediately spot that there IS a DNS query monitor configured. Why TF didn't it go off? It shows 100% uptime! I open up the settings.

Target: localhost.

🙈🙉

So a million years ago when I had just joined the air force, I was stationed in Southeastern turkey, and we were given prototype LAN setup. We had nine workstations scattered across base running unix, I don't remember which type.

The problem is that our official tech support was in germany, and although these machines could communicate with each other on base just fine, usually, it was truly a local network.

This meant that the people in Germany could talk you through a problem on the phone, assuming the phones worked that day. The power and telephones went out fairly regularly on base. But if it was something where they needed to be there in person they were about 2 days away, assuming one day to identify the problem, figure out they couldn't fix it, get permission to travel, fly down to the nearest city, get a rental car, and drive to base.

My boss decided, sensibly enough, that it would be useful if we had some sort of ability to troubleshoot problems on our own, so they paid for me to take a 3 week crash course in Unix system admin that GSA taught outside of Kansas City.

So I was about as competent at Unix sysadmin work as you would expect from a guy with a liberal arts degree and 3 weeks of training.

But never fear, new users in this case turned out to be even more problematic than my novice self

One night about 2:00 a.m. I was awakened by pounding at my door, the telephones were out again so it turned out that the night watch officer from the J2 drove to my dorm room to wake me up.

I'm pretty groggy at this point but I get dressed and go in. And he tells me that he had accidentally unplugged the workstation while it was in the process of booting up and now it wouldn't boot.

My first question, given that the power went out all the time and consequently we had uninterruptible power sources for each workstation, was to ask him if there was something wrong with his UPS

"No," said, it couldn't be a problem with the ups, because he had given it away two days previously. There were some people visiting the main base from a fob that we had just across the Iraqi border in the Kurdish area, and when they were visiting they told him that they didn't have anything for their own computers, so instead of directing them the communication squadron or logistics like a normal human would, he just gave them UPS from the watch desk.

"But don't worry, I got a hand receipt for it.". Proceeds to tear out a page from his notebook with illegible writing on it, other than the serial number. So I resolved to just hand that one over to my boss and let him figure it out.

So then I turned to solving the actual problem. In my groggy mental state, I could still remember to run fsck on the drive. It takes me about 20 minutes to get everything reset, partially because it's a slow hard drive and partially because I'm new and can only vaguely remember what I'm supposed to be doing

I get everything finished, and I am rebooting the machine so he can log back in because these are really slow it takes about 10 minutes. And this guy is nervous because he's embarrassed that he had to wake me up, he's embarrassed because he finally realized that he gave away hardware without authorization to do so and that there are may or may not be consequences (there were not),

So he's jabbering nervously at me randomly while the thing is rebooting and at some point he decides he wants to tell me how he accidentally unplugged the device while it was in the process of booting.

And he decides to use a prop, in this case a vacuum cleaner. He explains that night shift has to vacuum the watch floor every night, and he was vacuuming underneath one of the desks when it happened. As he does this he slams The vacuum cleaner underneath the desk in which the Unix workstation is plugged in. The vacuum cleaner hits the plug and rips it out of the wall, just like he had done earlier that evening

Because there is no UPS, the exact same thing happens. The box shuts down mid boot, won't reboot, and now I have to run fsck on it again. At least this time it took me only about 15 minutes to go through the process because it was fresh in my mind

That young Captain retired as a brigadier general. I sincerely hope he got smarter as he got older

A customer reported problems with accessing our web site so, fearing fail2ban had accidentally blocked their IP address, I pointed them to https://whatismyipaddress.com/ and asked them to tell me their IP address (as their mail headers weren't being helpful).

The reply:

> I'm a bit loathe to give out my IP address. - I've heard (rightly or > wrongly) that unsavoury things can happen when one's IP address gets out in the wild! Maybe I'm needlessly cautious, but I'm a bit paranoid about that sort of thing!

Despite this he's already gone to the web site I'd suggested and got his IP address from there so that web site knew it ... but he wouldn't tell us.

I'm off to find a wall to bang my head against.

I work for a medium size IT ServiceDesk company, who supports a large multi-national company.

The client company provides primarily laptop, the Windows OS security is customized for tight security and they do also use sites and account from vendor companies.

Not sure why but a certain part of our users, the phrase "this is out of our scope" or "we are not the correct support for this" is just a suggestion or its not true.

I got a call few months back about unable to log in to the citrix site cause the pingid is still on his old device and unable to authenticate. I then proceeded to unpair his device but I saw no devices registered or paired, so I went back to him and informed there are no devices paired to your account but insist there is so I then remote in to the laptop and asked to replicate where is he going.

I saw and its a 3rd party site and the PingID is registered in their end not ours, I then informed him that this is not within our scope and you need to contact that company helpdesk to reset your account or mfa on their for you to pair up your new device. He said that he called in their helpdesk the was referred back to us, I then asked to reach out again and say exactly that they need to "unpair my device from my account" but not sure what went on in his head and just said to them that he is unable to login from our from our company account then called back again to us.

I then suggested to send them an email but then said "I don't know what to say in the email" also now refusing to call back to the 3rd party helpdesk since he is unable to login using our laptop and refusing to believe the pingid is registered on their account and that we don't handle their accounts.

Went to my team lead for advice and suggested I send the email to the 3rd party helpdesk then cc the user on the email. This was resolved with the 3rd party support reset their PingID account.