u/RichSNJ

I have a client who is still running Exchange 2010.

Wanting to get them out of the stone ages, last week I started testing the migration from 2010 to 2016. I copied 2 of the domain controllers, the exchange server, and a workstation, and added them to a private Hyper-V network and ran some scenarios building a new server, prepping for and installing 2016, and moving mailboxes etc.

I'm not really sure what happened, but somehow I really screwed up. While I still don't understand how it could have worked so successfully from my testing private network, somehow on my test run my new Exchange server must have been on the production network instead of the private. I built it, moved most of the mailboxes to it, was super happy with it all and then I reset it to a checkpoint to run one more scenario before I was ready to do it for real.

This was mother's day, so I kicked off the install on the server and left the house. Got several calls while I was gone. Came back to find that the real world had been modified on the last run instead of the private network, and the server that I had transferred everything to was gone (back to the baseline checkpoint). Also, the Active Directory was prepped, which is I assume where the system mailboxes are created.

I spent that night getting the 2010 exchange server back from backups, however I am currently in a state where the Active Directory has already been prepped for 2016 and still thinks there is a 2016 Exchange Server on the network.

I still need to get this pig upgraded, but things are such a mess right now. I have been running various scenarios in my private test environment trying to manually clean up active directory using ADSIEdit, which I assume works, but I think that the cleanest method is to do the server recovery install for the missing server and then uninstall it.

I've done this off-line twice, and it is pretty clean, but before I can uninstall, I have to delete the database (which is essentially empty), but it still has all of the system mailboxes assigned to it, so it won't let me delete it.

Should I:

1.) Remove all of the 2016 system mailboxes and remove them from Active Directory (arbitration, auditlog, monitoring, federated etc) and run preparead again before I reinstall? Will this essentially be like starting from scratch and the installation will re-enable them etc...?

2.) Disable or remove them to the point where I can delete the database, but leave them all in AD, then either run preparead or not and hope they all wind up in the new database where they're supposed to be? Would I have to do the manual enabling of them in either of these first two scenarios?

3.) Move them to the Outlook 2010 server? I have never tried to move a newer exchange systemmailbox to an older exchange server, I don't even know if that's supported...

This is giving me an ulcer, so any help or advice would be appreciated.

Also, while the server recovery / uninstall method seems to work pretty well, both times I have tried doing this and reinstalling has resulted in an environment where the EAC works, but the EMS starts and doesn't have any exchange cmdlets loaded automatically. Not sure what that is about, but hoping it works when I do it for real in the production environment. I'm going to try and rename my new exchange server on my next text run to see if this mitigates that and whether I can find any trace of the old name in ad anywhere which might be causing an issue.

Last week I was doing upgrade testing on our Exchange Server. I imported copies of 2 domain controllers, the Exchange Server, a new exchange server, and a workstation onto a Hyper-V server and isolated them all on a private network. I have done this countless of times over the years / decades that I've been using Hyper-V.

I did the schema and domain preps on the new server and a full install of Exchange - all went with no issues. I then moved my mailbox over to the new server and had no problems. The CEO of this client has a 76 gig mailbox, so I moved that overnight and the next day it was fine. I believe it was Friday night and I decided to move over all of the mailboxes that didn't have online archive files, I kicked it off and went to bed.

Saturday morning I checked on it and everything had gone fine.

Keep in mind this was duplicate VM's all in a private Hyper-V network. The actual real environment was running with no issues in the real world. Several people accessed mail in the real world during all of this.

I didn't do anything else that day, Sunday I decided to run it through again, so I restored all of the Checkpoints on all of my test machines and started doing the install over again. I redid the schema and domain preps and kicked off the installation. It was mother's day, and I left to take my mom out, and while I was driving I got notified that people weren't getting any mail...

When I opened a computer, I logged into the live exchange server and everything looked ok, but when I logged into a workstation and tried to open Outlook, it wouldn't open. I tried to open Outlook Web Access and it told me that I needed to open my mailbox using the name of the server I was running in the test environment....

In short, somehow that I don't understand, what I did in the test had corrupted the real world. All of the mailboxes I moved were now showing as being on the new server, which to my knowledge had never been on the network. I had restored the checkpoints on all of those servers, so I couldn't even examine them to see what could have happened, and all of the mailboxes were gone.

I spent the rest of the long long night getting things back as best as I could, but now I am faced with the problem of wanting to test this upgrade again, and I'm afraid to do it now.

I have been trying to download the servers to my network here this week, but it's well over a terabyte, and it just hasn't finished yet. I really need to upgrade the Exchange server this weekend and I really want to run more testing, and I don't know if my downloads will complete.

So... Can anybody think of anyway this could have happened, and more importantly how to make sure it doesn't happen again. Like I said, I've been doing testing like this for years and years and I've never had anything like this happen. How could it be at all possible that the changes I made in the private network got through to the public network?