Active Roles - The connection with the remote endpoint was terminated

Hi

We are running Active Roles Server via 2 web servers which connect to 2 backend application servers, all running Windows Server 2016. Every 2-3 days we're getting an error on one of the web servers that "An error occurred during the last operation. Error: The connection with the remote endpoint was terminated" . The web server then needs to be rebooted for the site to connect and work successfully

Both web servers have identical .Net, IIS, NIC and app settings on them but only 1 is affected. Resetting IIS does not bring the access back and occurs no matter which browser is used, accessing Active Roles via the mmc directly to the app server that the web server uses works without issue

Any pointers would be appreciated

Thanks

  • It's possible that the Kerberos delegation setup for that box has "gone bad".

    See this article.

  • The other app and web servers with the same config and SPN accounts are working without issue though, if it was a kerberos / spn issue on the service account would a reboot of the server clear the issue?

  •   Have you looked at the event logs on the IIS host that is misbehaving to see if any errors are being thrown there?

  • Hi  

    You say you're seeing this issue every 2 or 3 days?

    If so, I would initially rule out any GPOs (user rights assignment/restricted groups) issue, as I'd expect the GPO re-application and/or Kerberos token refresh to have a shorter internal than 2 to 3 days.

    It may be worth checking the "Application Pool" recycling settings, as from memory that's usually by default set to an interval of every 29 hours (1740 minutes). You could test this by performing an recycle of the app pool for the ARS website on the affected server.

    It would also be useful to understand if your ARS deployment if configured to use constrained or unconstrained delegation, and what browser are you using when receiving the error.

    Its also worth confirming that both IIS servers (if using a load balancer and a single URL to access each server) have the same setting for the App Pool (Advanced Settings), specifically that:

    • The "Identity" is set to the same service account that the URLs SPN has been registered to
    • The "Load User Profile" is set to true

    Also that the Configuration of the ARS web site(s) is set correctly:

    • system.webServer/security/authentication/windowsAuthentication
      • useAppPoolCrendetials = True

    If you can check these, and let us know how your system is configured, and the existing settings for the app pool recycling, we might be able to trace the issue through. The information i'd be interested in are:

    • A list of all the SPNs registered for your Active Roles Administration Services service account
    • A list of all the SPNs registered for the SQL server hosting the Active Roles Administration Services
    • The details from the delegation tab in ADUC for the IIS Application Pool Service account used for the ARS Web sites (which should show entries for each Administration Service, both as FQDN entries and netBIOS entries)
    • The details from the delegation tab in ADUC for the Active Roles Administration Service Service Account (which should show entries for the SQL Server, both as FQDN entries and netBIOS entries and with and without instance ports)
    • Application Configuration
      • App Pool Identity
      • Load User Profile
      • Recycling settings
    • Web site configuration
      • Authentication settings, which are enabled and disabled
      • What providers are enabled for Windows Auth, and how are the advanced settings configured
      • In configuration editor what is the value of useAppPoolCredentials (as listed above), plus also the setting for useKernelMode

    Kind regards

    Stu