Optimize Synch Editor - Reduce (remove) Get Object During Synch

Hi All,

I'm working with a JDBC connector and I'm synchronising UNSAccountBInUNSGroupB with a target table of similar structure. I have around 18k records to process - right now I'm just reading them in.

My high level understanding of the sync engine is that it will fetch all objects, run through matching, then generally will run a get single object on each record to get the full property list. My mapping is fairly simple, I have two key resolutions which resolve UID_UNSAccountB and UID_UNSGroupB to their relevant ObjectGUIDs, and they are then mapped direct to the target fields which accept those GUIDs. 

I've noticed that my sync is taking around 30mins to complete and have noticed also that fetching a single object via the target system browser takes roughly 100ms. While I know the target DB is performing poorly (I will look into that with the owner), is there a way I can force the sync engine to pull the full object and property list up front, thus eliminating the need for single object reads. If I could do this, it would only take a few seconds to get the full object list rather than ~30mins (18k * .1 / 60).

I've tried playing with some of the advanced settings on the schedule (particularly upping the reload threshold for properties) and also extended settings on the workflow step (upping performance/memory factor to use more RAM) which I was hoping would do the trick. Looking at the memory usage of the job in the job server UI (executing instance view) the memory usage behavior appears the same no matter the settings.

All of that said, I'm after help/guidance on how (or even if at all) I can guarantee single object reads do not occur - or at least drastically reduce them.

Thanks,

Glen.

Parents
  • Hi Glen,

    what version of OneIM are you using?

    And some more questions:

    1. What type of OneIM connector are you using? As JDBC is not supported directly, I assume the ADO.NET connector?
    2. How many properties do you map for your object and what is the setting of the reload threshold at the startup configuration?
    3. Your target system schema does not contain any virtual attributes or does it?
    4. What are the settings of partition size and bulk level at the startup configuration?
  • HI Markus,

    Version is 8.1.4.

    1. ADO.NET generic.

    2. Just the two as mentioned: UID_UNSAccountB <-> UserId and UID_UNSGroupB <-> RoleId - the DB has app specific ids contained in its fields and I resolve them on the OneIM side in virtual properties to the relevant UIDs. Reload is maxed to test - 999. Was defaulted (I think, I wasn't the first to use this project) to 8.

    3. There's one virtual prop on the target side which is just the two ids concatenated to use as a primary key for each record (it is not mapped or used to match though). I can remove that actually and just use a compound key (of userid and roleid) if reducing virtual attrs helps. I'll do this now and see how it goes....

    4. I maxed those two out as a test (10240) and it still doesn't seem to use more RAM. 

  • Did you run the "Synchronization analysis report" from the Help menu in Synchronization Editor to verify the load times for a small list, and a single object?

    Using the target system browser is not a good indicator at all times, because it always uses the single load object scheme for a single object.

  • It shows a 66ms average load time. WHich when extrapolated is (as mentioned) quite a long time.

  • Can you post all values or attach the complete report, please?

    The ADO.NET connector is using a local SQLite database where the results of your custom SQL statements will be stored to be able to fetch single values at all.

    So if 66ms it is, this should be the time to fetch the data from the SQLite tables.

Reply
  • Can you post all values or attach the complete report, please?

    The ADO.NET connector is using a local SQLite database where the results of your custom SQL statements will be stored to be able to fetch single values at all.

    So if 66ms it is, this should be the time to fetch the data from the SQLite tables.

Children
No Data