Very exited to see announcement of identity recovery with ad forest. Currently are Veeam house, but Rubrik have been making efforts with our management to move and we hope it is competitive to Rubrik as we don’t want to have to change .
I wanted to share our experience as a VSP after upgrading our Cloud Connect environment to Veeam 13, partly to see whether other providers are seeing similar behaviour, and partly because at the moment it has been a very difficult few weeks operationally.
To be clear upfront, we did not initially intend to upgrade as early as we did.
As a service provider, we tend to prioritise stability over moving immediately onto major new versions. However, a growing number of customers began upgrading their own VBR servers due to the security vulnerability concerns and advisories around older versions.
As many providers will know, once customers upgrade ahead of the provider platform, Cloud Connect compatibility becomes an issue. Even though Veeam does warn customers during installation, many understandably proceed anyway due to the security messaging, and we eventually reached a point where enough customers had upgraded themselves that we effectively had to move the provider platform forward sooner than planned.
Unfortunately, since upgrading, we have experienced a sequence of different issues affecting various parts of the platform.
The first issue we encountered involved infrastructure components apparently presenting incorrect or mismatched identities/certificates after upgrade. This seemed to lead to transport instability and certificate validation problems between components.
After working through that, we then began seeing .NET SSL-related issues within VSPC.
Separately, we also experienced periods where Veeam ONE activity appeared to place significant load on the Veeam RESTful API, which impacted provider-side administration tasks such as editing tenants, managing the platform and even running of PowerShell scripts.
Most recently, we are now dealing with a large amount of Cloud Connect retry activity associated with Continuous Copy Jobs.
Initially we saw intermittent failures, but over time this developed into very large numbers of repeated retries across multiple tenants. While investigating, we began noticing some chains with extremely large incremental depths, and from our observations it appears possible that retention/consolidation may not always be occurring as expected in certain scenarios.
The concern from our side is that these large chains then appear to contribute to ongoing retry behaviour, with jobs repeatedly attempting to process throughout the day.
Storage consumption has also become a concern. In some cases, triggering an Active Full appears to temporarily restore job operation, however if the previous chain is not cleaned up as expected, storage usage can grow rapidly due to older incremental chains remaining alongside new fulls and new incremental growth.
This evening our second ticket was close and re-opened in another ticket as a new issue (each time we fix something a new ticket needs opening for the other parts), and we were provided with a potential workaround specifically relating to the recurring terminated/killed session behaviour we are currently investigating.
The proposed workaround from Veeam R&D was to add the following registry key on the customer-side VBR server:
Path:Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name:ProcessVirtualizationExecutionsPerManager
Type:DWORD
Value:1 (Decimal)
The difficulty for us operationally is that, as a provider, we do not necessarily have administrative access to customer VBR servers, nor can we realistically perform out-of-hours reboots or manual changes to customer environments while active backups are running and it would need prior customer agreement. So sadly its going to be another day before we get to see if this will work or not.
We were advised that this workaround is currently the available recommendation from R&D and that a more permanent fix is expected in a future product version.
I also want to be fair regarding support. The individual engineers we have interacted with have generally been polite and professional throughout. However, the overall process has been quite difficult to navigate during an active provider-side incident.
Because the issues have manifested in different ways over time, cases have sometimes been separated or recreated, which has unfortunately led to repeated log uploads, repeated explanations of earlier findings, and delays while context is re-established between teams and regions.
There have also been occasions where we have remained available outside normal working hours after being asked whether we could continue troubleshooting with teams in other regions, but communication and handovers have not always flowed as smoothly as we would have hoped during those periods.
From a provider perspective, the overall operational impact has been significant:
At the moment we are still actively working through these issues with support and trying to stabilise the platform.
I would genuinely be interested to hear whether any other VCSPs or Cloud Connect providers have experienced similar behaviour after moving to Veeam 13, particularly around:
Hopefully this helps start a constructive conversation between providers experiencing similar issues.
I am confused about how immutable backups work in Veeam. We use wasabi object storage. The bucket has object lock enabled. I want 30 days of immutable backups.
The backup job has 30 days retention. The host is either proxmox/ hyper-v or vsphere esxi. We have a mix of v13 (windows version and appliance) and v12. The biggest issue is that i have is that the onsite backup has a size of 1.6tb and the offsite (wasabi) is 3tb
Could it also be a v12 issue?
Hello,
there is a HyperV Host Windows 2025 with approx 5 Windows VMs
and there some automatic Veeam Recovery Snapshots (due to daily backup). Is there a way to avoid such Snapshots?
I know, manually deletion via Powershell is required.
Here is my network. I inherited it from previous team. BKMGR01 is not domain joined. Trying to make it work the best I can for Direct Storage Access. I might be thinking about this wrong. NIC one being on the 29 network i thought allows it to speak to other hosts since VMWare hosts are on that network. My Second Nic on the Physical server has 2 IPs on it since the SAN is on 2 networks. I could make that 1 network if I wanted if that 2nd IP address is causing some issues. I am open to suggestions. My backups are slow.
IDEAS!
If I moved the QNAP to the 10GB backbone switch I could run 2 10Gb connections from QNAP to that switch since there is a free one. Would that make more sense.
I am open to other ideas as well.
Thought I’d throw this up here because this has bitten me a few times over the years now, and every single time I end up finding absolutely nothing useful online about it.
Had it happen again today during a Veeam upgrade and finally thought, right, I’m posting the fix in case it saves somebody else the headache.
The issue:
You run a Veeam install, upgrade, or Service Provider Console upgrade.
Installer says prerequisites need installing.
It asks for a reboot.
You reboot.
Installer comes back.
Same prerequisite message again.
Rinse and repeat forever.
Nothing actually installs.
No obvious errors.
Just stuck in a stupid reboot/prerequisite loop.
What made this especially frustrating was Windows itself isnt reporting any pending reboot state's. All the usual checks look clean.
In every instance, the culprit has been a stale Veeam RunOnce registry value that never got cleared.
This was the value:
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce\VeeamWizardEngineNeedReboot
More specifically:
Key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce
Value:
VeeamWizardEngineNeedReboot
Deleted that value, reran the installer, and the upgrade immediately carried on normally.
No more reboot loop.
Obviously usual disclaimer applies:
Be careful in the registry, export the key first if you’re unsure, etc.
Not saying this is the fix for every prerequisite issue, but if:
…definitely check that key before you lose half your day to it like I did.
Hopefully helps someone in the future searching this at 2am while questioning their life choices.
I work for an MSP and we have been moving our clients to Veeam from Storagecraft/Arcserve ShadowProtect. The move has been bumpy along the way as I try to wrap my head around all the moving parts to Veeam vs ShadowProtect. We have a repository in our data center where we host our client's backups using CloudConnect. We upgraded our storage and moved our client's backups to a new repository. Things went well, or so we thought. For some reason, when we attempt to create a new tenant, we get an error that Veeam support believes is due to stale records in the database. I've had multiple scheduled Zoom calls and I feel like the support engineers are always in a hurry to get off the phone with me. They'll have me try one thing, which doesn't work, and then tell me to upload the log files so they can further review. Days will go by without any communication. It's been nearly 3 weeks, and the case was forwarded to RnD. I will check in with the support engineer every couple of days, and I'm told his manager knows about the situation and they are waiting for RnD. Meanwhile, we have a few clients that are not able to backup to us, and I can't add any new clients. While Arcserve support was never stellar, they stayed on it and worked with you until it was resolved.
I recently installed the linux appliance for VBR 13 and am unable to perform file level restores.
When performing a restore I get the error "Failed to log in to "Vsphere Server", port 443, user "Domain.com\Account", proxy srv: port:0".
I am currently working with support and any help on this would be greatly appreciated and rewarded with copious internet points
Good Day,
I have a new standard OL9 installation that I use as a VEEAM Repository. The services that are installed are unconfined. I have been asked to fix that. I have fixed a few other services, but they had SELinux Policy modules - I don't see one for VEEAM. I also don't see any guidance.
Can you point me to a resource that can help? I feel that VEEAM must have an answer to this. We are using VEEAM 12.
Just a piece of mind question but has anyone ran into any issues running the Veeam Linux hardened repository on AMD servers? I cant think of a reason it would cause any issues but my director is just seeking clarity.
We currently run VBR on a windows server and connect to a remote SQL server. Im interested in either doing one of these two things and wondering what has been experiences so far with either.
Objectives:
I just finished setting up our Veeam Enterprise Manager server and wanted to serve it as a website thats only accessible to our office WAN IPs. Does anyone know how one would accomplish this or if it is doable? I see the website in IIS and I do have a valid wildcard cert but support told me they could not help with design so I am unsure what the proper way to serve this webpage would be. Any assistance would be appreciated.
Hello
any difference between a "standard linux" like almalinux and for example alpine linux from a backup perspective?
data integrity and security are very much requested here, the number of docker guest will probably be 4 on exactly 1 docker host, the docker host is vmware-esxi right now but will probably be something else in 6 months.
these 4 docker guest are a web frontend, a load balancer, a small database (probabyl mariadb or postgres), and a filestorage thing,
all together 100 GB or less.
do i need to stop the docker containers for a very reliable backup? what about restore?
im very new to this stuff.
thanks & bye.
