Long Service Startup Times

Greetings! 

So curious to see if this has happened in other environments. We have three 2012 R2s servers each running their own ARS node that connects to one SQL database. They all work fine except when we have to reboot them for maintenance, one takes two hours to finish the startup while the other two take two minutes. All three use the same service account to connect. All three servers *look* identical, but clearly not. Has anybody else experienced this before, or are there any flags that I could check to see if there is a difference?

Any feedback will be greatly appreciated. Thank you.

  • Hi Julia,

    Are you staggering the starting of the services or starting them all at once? The proper way to restart admin services when there are more than one is to start one and wait until it's full up and you're able to log into it before moving onto the next one.

    The other thing could be physical location if that one server is really far away on a slow connection to any DC's for managed domains or even the SQL database, it can take a really long time to communicate with the DC's during startup. This is seen often in regionalized configurations. If it's the SQL database, you could look into setting up another SQL host in the same location as that host and configure database replication. Similarly if it's the DC's, you could configure that service to connect to local DC's for the managed domains if it's connecting to DC's that are in another part of the world.

    I hope this gives you some things to check.

  • Thanks for the feedback! 

    I *wish* it was physical location, but all three ARS nodes plus the SQL server are located in the same area, which rules out physical distance issues. And all three services connect to local DCs (which are in the same location as the ARS nodes/SQL server). It's just annoying since there doesn't seem to be a reason for this one node to take waaaay longer than the other two.

  • What about the startup sequence of the servers?

    Outside of that, someone would have to have a look at the event log. Speaking of that, do you maybe have debug logging enabled on that server? Debug logging can slow things down pretty drastically so that would definitely be worth checking.

  • I would also be curious:

    A) Is there a connection to an Office 365 tenant in the mix
    B) Are there any scheduled tasks, dynamic groups or group families that only the one node is responsible for?
    C) Which of your admin services was the "first" one deployed originally?
    D) Are there any differences with installed software and/or patch level (thinking things like Powershell and .Net in particular)?
    E) Have you checked the SQL connection for latency from that host?
    F) Is that host running some kind of endpoint protection that the others are not?

  • For the sequence, we wait until one is up before booting up the other one. 

    Did check the logs in the Configuration Center, and all are 'Disabled'. Would there be other spots to check for logging?

  • A) Negative

    B) It is in charge of one scheduled task, but considering the others have more on their "plate", that seems unlikely...I mean, a schedule task isn't supposed to run before the startup is built right? Or can the scheduled task kick off during the midst of a startup operation? (Which then I could see being a delay.)

    C) The first one deployed is this slow one. The other two were deployed/installed about three weeks later.

    D) Checked Powershell version and .NET and they are identical.

    E) Can ping the SQL server from the ARS host and that seems fine.

    F) All three have McAfee/HIPS, so if that was it, I would think they all would be slow/start up time would be the same.

    Thanks again for all the suggestions! I do really appreciate it.