u/mazvazzeg

Hey,

I could start this with "I built a ..." just to trigger the AI-slop-sensitive part of the community :P - but I won't. I just wanted to show you something which I'm using for troubleshooting/debuging Hyper-V.

hvtop / VMs

Coming from VMware enviros, I really missed the esxtop functionality on Hyper-V. If you don't know it, it's a TUI driven performance monitor for ESXi hosts. Fast, very specific, but gives you the most important metrics without any GUI or webservice or plugin requirements.

I can't find anything close enough for Hyper-V (or Windows, for that matter) - you'd need to juggle Hyper-V Manager, Performance Monitor, Task Manager, Resource Monitor and who knows what else to check what's killing a host's performance.
(I know, WAC will do it for you with a nice GUI, but that's exactly what I didn't want).

While I am quite handy with Powershell, and I have crafted my basic reporting PS scripts to get data similar to esxtop's, I soon realized PS isn't the right tool for this type of data/metric gathering, it's just too slow to give reasonably precise feedback about what's happening.

So. Enter Codex.
(Now you can boo me. Really. I get it. But if you don't want to use it, then just don't. Of course you can still drop the usual AI-slop comment if that makes you happier. I don't mind.)

So hvtop now is a C# utility, able to check the basics (CPU/RAM/DISK/NETWORK), and some more detailed info as well. It can drill down to VM or Storage or vSwitch level for more information, and it works with Failover Clusters.

you can check it out here (open source):
https://github.com/mazvazzeg/hvtop

Hello,

ran into what looks like a Veeam bug yesterday, just want to know if anyone else has also experienced similar or this is something very specific to our environment.

It's a fresh greenfield Hyper-V Failover Cluster infra, and Veeam is running in a VM. Version 13, Community Edition.

(Don't want to make a detour here, but: the install ISO is 20 GB. Twenty. Gigabytes. Good god. The first version I worked with was the 5.0, and that was 200 MB. Anyways...)

Now the fun part: after configured everything (Backup Job, Repo, Applicaton Aware Processing), Veeam backups everything just fine - except the backup VM itself. Says can't make production checkpoint (snapshot), error 32770.

The closest thing I find in Veeam's KB is this article:
https://www.veeam.com/kb3137

Did all the the tricks but it just pointed to the VSS integration component error:
VSS Lost Communication. And on the guest VM, the Hyper-V Volume Shadow Copy Requestor service can't start (Error 1053).

Now the fun part: poking around this service, I found this in the service configuration of this service (HKLM\SYSTEM\CurrentControlSet\Services\vmicvss)

ImagePath REG_EXPAND_SZ %systemroot%\system32\svchost.exe -k VeeamHvVssGroupstemNetworkRestrictedRedirectionGuard -p

Which is odd. Checked in other VMs where VSS just works fine:

ImagePath REG_EXPAND_SZ %SystemRoot%\System32\svchost.exe -k VeeamHvVssGroup -p

Hahh. Copy + paste this to the Veeam server. Lo and behold, service can start, VSS production checkpoint works, VM AAP Backup runs fine.

But how that garbage ended in the registry key in the first place? It seem to me that something (Veeam installer?) tried to modify this key from the default, but instead of replacing it with "VeeamHvVssGroup", somehow it just overwritten the first few characters? Don't know actually, just trying to understand what happened here.

Anyone else seen this behavior? Or this is just me?

hvtop - a TUI Hyper-V telemetry / performance monitor