AD Provisioning fails with: Unable to execute method (Insert object) for object (Xxxx) because not all mandatory properties are defined.

My AD Account Provisioning has stopped working.  It always fails with the error "not all mandatory properties are defined".  It is complaining about "cn, objectClass, sAMAccountName".

To eliminate moving parts, I go to the Sync Editor.  I go to the Browse One Identity, and click on one of the ADSAccounts that failed to create.  If I look at the properties, I can't find one single problem.  "cn" looks perfect.  "objectClass" is just "USER", "sAMAccountName" is only upper-case characters and does not exist in the directory.  Nothing to see.  Click [Test Provisioning] and instantly it tells you "not all mandatory properties are defined".

The Sync project is out of the box ...next...next.. next... save.  I put the Templates for "cn, objectClass, sAMAccountName" back to their default scripts.  I saw somone else on the forum suggesting that this could be caused by AD permissions.  I just want to see this work, so I made my Service Account "Enterprise Admin".  

This sysc project has worked perfectly for about 3 years, then two months ago it stopped working.  Not a clue what has changed.  I'm running 8.1.4 with all One Identity as well as Microsoft patches applied.

Here is the full error output...

[1777018] Error executing synchronization project (Active Directory Domain (DC=IDS,DC=LOCAL))'s workflow (Provisioning).
at VI.Projector.Projection.ProjectorEngine.<ExecuteAsync>d__3.MoveNext()
[1777124] Error executing projection step (person) of projection configuration (Provisioning (Provisioning)).
at VI.Projector.Projection.ProjectorEngine.<ExecuteAsync>d__3.MoveNext()
[1777219] Error executing synchronization step (person)!
at VI.Projector.Projection.ProjectorEngine.<ExecuteAsync>d__3.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
--- End of stack trace from previous location where exception was thrown ---
at VI.Projector.Projection.ProjectionStrategy.<ExecuteStepAsync>d__3.MoveNext()
[1777376] Unable to execute method (Insert object) for object (Ealswid Zeyneb) because not all mandatory properties are defined.

Missing are: cn, objectClass, sAMAccountName
at VI.Projector.Projection.ProjectionStrategy.<ExecuteStepAsync>d__3.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
--- End of stack trace from previous location where exception was thrown ---
at VI.Projector.Projection.ProjectionStrategyBase.<OnExecuteStepAsync>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
--- End of stack trace from previous location where exception was thrown ---
at VI.Projector.Projection.ProjectionStrategyBase.<_ProcessDifferenceSetAsync>d__14.MoveNext()
at VI.Projector.Projection.ProjectionStrategyBase._ExecuteMethods(IProjectionStep step, SystemObjectMatchingSet matchingSet, ISystemConnection connection, ISchemaMethodAssignment[] methodAssignments, IList`1 systemObjectPairs, SystemMappingSide side, ProjectionDirection projectionDirection, Boolean forceReloadObject, Boolean ignoreQuotas)

Crash report

2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemObjectData Creating SystemObjectData based on entity (%InternalName% [%CCC_PersonId%]) columns (UID_Person, XObjectKey).
2021-02-24 09:25:05 SystemConnector Result: State (Success) Objects (1728) Failures (0)
2021-02-24 09:25:05 GenericPool Release item: IDS-DB01\D1IM_8
2021-02-24 09:25:05 SystemConnection QueryObjects result (out of scope): 1728 system objects.
2021-02-24 09:25:05 StopWatch Querying objects of schema class (Person_Master). done in 57ms.
2021-02-24 09:25:05 SystemScope ...1 objects are in scope.
2021-02-24 09:25:05 StopWatch Filtering scope (1f315825-6b26-4548-b895-04a3362c55f6)... done in 63ms.
2021-02-24 09:25:05 StopWatch Querying objects of Employees (Person) where UID_Person=12659588-1757-49d3-ac51-ee015f626fd6. done in 91ms.
2021-02-24 09:25:05 SystemMappingRule Mapping applied.
2021-02-24 09:25:05 ProjectorEngine Cannot execute method (Insert object) on object (Ealswid Zeyneb) because not all mandatory properties are defined! -> Failure
Missing: cn, objectClass, sAMAccountName
2021-02-24 09:25:05 SystemConnection ProcessContext has changed from (0028327c-a980-408b-b9f7-9bb38e3b89a9) to (62ef2197-682c-488f-ac09-c66325548689).
2021-02-24 09:25:05 SystemConnection ProcessContext has changed from (0028327c-a980-408b-b9f7-9bb38e3b89a9) to (<null>).
2021-02-24 09:25:05 SystemConnection Leaving update phase.
2021-02-24 09:25:05 SystemConnection Leaving update phase.
2021-02-24 09:25:05 SystemConnector Leaving update phase.
2021-02-24 09:25:05 StopWatch Executing step (person)... done in 401ms.
2021-02-24 09:25:05 StopWatch Time consumption for (SQL): 6ms (1.51%)
2021-02-24 09:25:05 StopWatch Time consumption for (Cache): 0ms (0.03%)
2021-02-24 09:25:05 StopWatch Time consumption for (Session): 0ms (0.00%)
2021-02-24 09:25:05 StopWatch Time consumption for (Entities): 17ms (4.29%)
2021-02-24 09:25:05 StopWatch Time consumption for (DatabaseReader): 0ms (0.00%)
2021-02-24 09:25:05 StopWatch Time consumption for (One Identity Manager (IDS-DB01\D1IM_8)): 211ms (52.64%)
2021-02-24 09:25:05 StopWatch Time consumption for (Active Directory Service (Root DN dc=IDS,dc=local, Server ids-dc01.ids.local)): 28ms (6.92%)
2021-02-24 09:25:05 StopWatch Time consumption for (ProjectorEngine): 1ms (0.19%)
2021-02-24 09:25:05 StopWatch Time consumption for the rest: 138ms (34.41%). (This result might be affected by multithreading!)
2021-02-24 09:25:05 ProjectorEngine Performing finalization tasks.
2021-02-24 09:25:05 SystemConnection Leaving simulation mode.
2021-02-24 09:25:05 ProjectorEngine Unregister journal recorder.
2021-02-24 09:25:05 ProjectorEngine Reset variable set.
2021-02-24 09:25:05 SystemConnection Disconnecting connection One Identity Manager (IDS-DB01\D1IM_8)...
2021-02-24 09:25:05 SystemScope Invalidating the entire scope (Scope). Force = True
2021-02-24 09:25:05 SystemScope Invalidating the entire scope (Reference Scope). Force = True
2021-02-24 09:25:05 SystemConnector Disconnecting target system.
2021-02-24 09:25:05 ObjectLog Closing session
2021-02-24 09:25:05 SqlLog Closing database session.
2021-02-24 09:25:05 SqlLog Closing database session.
2021-02-24 09:25:05 SystemConnection Connection disconnected successfully (One Identity Manager (IDS-DB01\D1IM_8)).
2021-02-24 09:25:05 SystemConnection Leaving simulation mode.
2021-02-24 09:25:05 ProjectorEngine Unregister journal recorder.
2021-02-24 09:25:05 ProjectorEngine Reset variable set.
2021-02-24 09:25:05 SystemConnection Disconnecting connection Active Directory Service (Root DN dc=IDS,dc=local, Server ids-dc01.ids.local)...
2021-02-24 09:25:05 SystemConnector Disconnecting target system.
2021-02-24 09:25:05 SystemConnection Connection disconnected successfully (Active Directory Service (Root DN dc=IDS,dc=local, Server ids-dc01.ids.local)).
2021-02-24 09:25:05 ProjectorEngine Flushing journal...
2021-02-24 09:25:05 ProjectorEngine Disposing process context.
2021-02-24 09:25:05 ProjectorEngine ProcessContext has changed from (5d7acaff-00e7-427e-8f85-4e27828fdd5b) to (<null>).
2021-02-24 09:25:05 StopWatch Provisioning (Ealswid Zeyneb) with configuration (Provisioning)... done in 1.53s.
2021-02-24 09:25:05 StopWatch Time consumption for (SQL): 32ms (2.11%)
2021-02-24 09:25:05 StopWatch Time consumption for (Cache): 0ms (0.01%)
2021-02-24 09:25:05 StopWatch Time consumption for (Session): 15ms (1.00%)
2021-02-24 09:25:05 StopWatch Time consumption for (Entities): 17ms (1.12%)
2021-02-24 09:25:05 StopWatch Time consumption for (DatabaseReader): 0ms (0.00%)
2021-02-24 09:25:05 StopWatch Time consumption for (One Identity Manager (IDS-DB01\D1IM_8)): 211ms (13.77%)
2021-02-24 09:25:05 StopWatch Time consumption for (Active Directory Service (Root DN dc=IDS,dc=local, Server ids-dc01.ids.local)): 28ms (1.81%)
2021-02-24 09:25:05 StopWatch Time consumption for (ProjectorEngine): 1ms (0.05%)
2021-02-24 09:25:05 StopWatch Time consumption for the rest: 1.23s (80.12%). (This result might be affected by multithreading!)
2021-02-24 09:25:05 ProjectorEngine Error executing projection step (person) of projection configuration (Provisioning (Provisioning)).
2021-02-24 09:25:05 VI.FormBase.ExceptionMgr One or more errors occurred.