I recently got into trouble with this useful feature and figured that maybe if I share my experience, I can save someone the headache I went through.
AD Connect Staging Mode was introduced to provide an active-passive HA design to AD Connect – critical for anyone using Microsoft cloud services with hybrid Active Directory identities. If you aren’t familiar with AD Connect, then you can read about it here.
To utilize the Staging Mode feature, you configure a primary AD Connect instance and an identically configured Staging instance. The Staging instance runs scheduled synchronizations (just as the primary does) with the difference being the Staging instance doesn’t actually export anything to Azure AD unless Staging Mode is turned off (which should only be done if the primary instance is decommissioned or otherwise unavailable).
So far so good, right?
As an added bonus of Staging Mode, if you are introducing AD Connect into an existing hybrid AD environment and you want to confirm what the consequences will be once you unleash it, there is a reporting mechanism providing you a preview of what would happen if you were to disable Staging Mode. Some refer to it as a “what-if” mode.
This is where our story gets interesting…
I’ve used Staging Mode many times in the past to confirm what I was about to do wouldn’t cause any harm in the way of adds/deletes of users/groups/contacts etc. and it always gives me a nice warm and comfy feeling to see that a catastrophe is not about to occur – I’m a big fan of warm and comfy!
Recently, I was involved with a project and had to re-sync an on-prem AD environment with Azure AD which was disconnected several years prior after completion of a hybrid Exchange migration. The client was finally ready to stop managing two sets of identities and wanted to implement an identity management and SSO solution, so these estranged directories needed to be reunited like a happy family once again!
We had a lot of identities to marry up and needed to determine which attributes should survive as well as which groups, contacts, and users should be synced, and which items were not going to be synced (such as distribution groups). Lots of fun PowerShell scripts to write here, folks! Maybe that can be another blog sometime.
In the final implementation stage, while running a Staging Mode report, I came across some legacy Mail Enabled Security groups in AD (which existed prior to the hybrid migration) which were going to update the Office 365 versions of these groups which were actively being used for email distribution lists throughout this global organization.
The group memberships were managed in Office 365 and the last thing we wanted to do was overwrite the memberships of these groups from the AD version of them as it would be waaaaaaaaay out of date. So, we deleted those groups from AD – which in my mind meant the Office 365 version of them would stay as cloud and un-synced. Sounds good, right?
Well, not quite…
AD Connect maintains something called a Metaverse which is a dynamic database of both AD and Azure AD directories and what the merged directory consists of (down to the attribute level on an ongoing basis, even in Staging Mode). So, since I deleted these groups in AD, AD Connect naturally decided I wanted them to be deleted in the cloud as well and it updated the Metaverse. Once I removed Staging Mode my assumption was that AD Connect would reconcile what is current in AD and Azure AD and do a fresh synchronization. What actually happened was that AD Connect already knew what its next order of business was after removing Staging Mode… pushing out whatever changes were queued up in the metaverse and DELETING THESE GROUPS in Office 365!!!
So much for Staging Mode being a harmless what-if tool! It was ready to replicate any changes processed once Staging Mode was removed, which in hindsight makes perfect sense. Sometimes what I think I know about how something works, distorts my ability to confirm how it actually works. Again, another topic for another blog!
Let’s get this cleaned up…
Since security groups are not a restorable object in Azure AD, I was forced to recreate those groups using data I had backed up at the beginning of the project. In the course of my career, I have learned to always plan for the worst while hoping for the best, so I indeed had a backup of all Azure AD group memberships – whew!
To make matters worse, I had paused the AD Connect sync schedule while doing this work and when I finally re-recreated those groups, I re-enabled Staging Mode and ran a sync and wouldn’t you know it, AD Connect once again was eager to delete those groups!
I just uninstalled and reinstalled AD Connect which created a new Metaverse and all was good from then on but once again Staging Mode to the rescue! I believe there is some irony here in using Staging Mode to help protect me from, well, Staging Mode itself!
After this ordeal, in chatting with an AD Connect SME at Microsoft, I was advised that instead of uninstalling and reinstalling I could have saved a few minutes and just cleared the connector spaces for both AD and Azure AD and essentially accomplished the same thing. But I haven’t tried this yet… please let us know if you have.
To do this, you would clear the connector space for both connectors (AD and AAD) by opening the Synchronization Service Manager, opening the Connections tab, right-clicking on each connector and selecting “Delete…” followed by “Delete connector space only”.
So, go forth and stage away but be careful and ALWAYS plan for the worst!
Want more insights like this delivered to your inbox? Make sure you sign up for Insentragram.
Enter your details to join Insentragram
ABOUT THE AUTHOR
Senior Cloud Architect at Insentra
A Consultant within our Cloud and Collaboration practice, Neil is responsible for working with customers to implement Microsoft Cloud solutions in Azure and Office 365 on both presales as well as delivery functions.
Neil enjoys over 20 years of diversified industry experience across a variety of technology solutions which aids him in being able to visualize all of the components comprising a finished solution including infrastructure services, networking, security, identity, application, n-tier architecture, availability, business continuity, disaster recovery, data governance, compliance and combining IaaS, PaaS, and SaaS services.
Prior to joining Insentra in 2018, Neil worked with a New York City based Microsoft Partner implementing Azure and Office 365 solutions in both cloud native as well as hybrid environments. In addition to performing presales and project execution, Neil excelled at working directly with customers on behalf of Microsoft as a Partner Seller (P-Seller) and Partner Cloud Solutions Architect (P-CSA) delivering envisioning and whiteboarding sessions as well as conducting technical webinars. Before becoming a consultant in 2014, Neil was on the customer side working as Systems Administrator, Network Engineer and IT Director.
Neil has recently relocated to Houston, TX after living his entire life in the New York City metropolitan area and enjoys a great family life, with a loving wife and two beautiful young daughters who are his pride and joy!
A SUMMARY OF KEY SKILLS:
• Migrating VM’s, applications and entire datacenters to Microsoft Azure
• Hybrid networking between premises, colocations and the cloud using VPN, ExpressRoute and SDWAN
• Leveraging the cloud for business continuity and disaster recovery
• Azure IaaS, networking, security and resiliency limitations and best practices
• Azure Active Directory (AAD) hybrid identity deployment and management
• Enterprise Mobility + Security (EMS) including Multi-Factor Authentication (MFA), Conditional Access (CA), Self Service Password Reset (SSPR) Intune device management and Identity Protection services
• Microsoft Cloud App Security (MCAS)
• Identity federation and Single Sign-on (SSO)
• Azure enterprise subscription governance
• PowerShell scripting and Azure Automation
• Office 365 management
• Active Directory Domain Services (ADDS) maintenance, troubleshooting and best practices
• Microsoft infrastructure technologies including Group Policy, Failover Clustering, Hyper V, SQL, RDS etc.
• General datacenter components e.g. servers, hypervisors, storage, switching, routing, firewalls, LB’s etc.
• Authoring architectural documentation and Visio diagrams