Troubleshooting a workflow running a script

Hi All,

We are trying to troubleshoot an issue with a scheduled workflow that runs a script.

The script uses the MicrosoftTeams PowerShell module to get a list of users with Teams voice enabled, it then updates a virtual attribute in ARS called edsvaTeamsVoiceEnabled accordingly.

The workflow job runs absolutely fine most of the time, but randomly the CPU on the ARS server will max out and not drop until we restart the service. When this happens the job is running although it completes with the normal timeframe (less than 45 seconds). After than point, we can see the in run history that the job still randomly terminate even though it appears to complete, this doesn't happen every time though.

We have moved the job between multiple ARS servers with the same result. We have also ensured all other jobs are not running on the same ARS server to confirm it's this one job that causes the issue. If we disable the job, we do not see the issue with the CPU spike.

We have run the script manually multiple times and haven't been able to troubleshoot the CPU issue.

We have included a transcript in the script, but this just shows the job completing successfully.

Has anyone got any tips on how we can troubleshoot this further?

We are running ARS 8.1.3, SP1

For reference, here's the script:

function Update-MSTeamsVoiceEnabledUsers() {
    $logPath = 'C:\ActiveRolesLogs\Update-MSTeamsVoiceEnabledUsers\'
    if (-not(Test-Path $logPath)) {
        New-Item -ItemType Directory -Path $logPath
    }
    $transcriptName = "Update-MSTeamsVoiceEnabledUsers"
    Start-Transcript -path "${LogPath}\${transcriptName}.$(get-date -f yyyy-MM-dd-HH-mm-ss).txt" -NoClobber
    try {
        $ProgressPreference = "SilentlyContinue"
        $ErrorActionPreference = "Stop"
        $appRegistrationCreds = 'CN=AzureAD Creds,CN=Library,CN=Script Modules,CN=Configuration'
        Invoke-Expression $(([adsi]"EDMS://$appRegistrationCreds").edsaScriptText[0])
        Import-Module MicrosoftTeams

        Connect-MicrosoftTeams @azureCreds | Out-Null
        [System.Collections.Generic.List[string]]$voiceEnabledUsers = 
            (Get-CSOnlineUser -Filter 'EnterpriseVoiceEnabled -eq $true -and LineUri -ne $null').UserPrincipalName
        $adVoiceDisabledParams = @{
            SearchRoot                       = "OU=Company Users,DC=somewhere,DC=local"
            LdapFilter                       = "(|(edsvaTeamsVoiceEnabled=false)(!(edsvaTeamsVoiceEnabled=*)))"
            DontUseDefaultIncludedProperties = $true
            IncludedProperties               = @("UserPrincipalName")
            Enabled                          = $true
        }
        [System.Collections.Generic.List[string]]$adVoiceDisabledUsers = (Get-QADUser @adVoiceDisabledParams ).UserPrincipalName
        foreach ($user in $voiceEnabledUsers) {
            if ($adVoiceDisabledUsers -contains $user) {
                # user found in the array, removing to save time on future lookups
                $adVoiceDisabledUsers.Remove($user)
                #"Setting edsvaTeamsVoiceEnabled to true for $($user)" | Out-File C:\teamsvoice.log -Append
                $params = @{
                    Identity         = $user
                    ObjectAttributes = @{edsvaTeamsVoiceEnabled = $true }
                }
                Set-QADUser @params
            }
        }
        $adVoiceEnabledParams = @{
            SearchRoot                       = "OU=Company Users,DC=somewhere,DC=local"
            LdapFilter                       = "(edsvaTeamsVoiceEnabled=true)"
            DontUseDefaultIncludedProperties = $true
            IncludedProperties               = @("UserPrincipalName")
            Enabled                          = $true
        }
        [System.Collections.Generic.List[string]]$adVoiceEnabledUsers = (Get-QADUser @adVoiceEnabledParams ).UserPrincipalName
        foreach ($adUser in $adVoiceEnabledUsers) {
            if ($voiceEnabledUsers -contains $adUser) {
                # user found in the array, removing to save time on future lookups
                $voiceEnabledUsers.Remove($adUser)
                # continue to the next user as account is correctly voice enabled
                continue 
            }
            #"Setting edsvaTeamsVoiceEnabled to false for $($adUser.UserPrincipalName)" | Out-File C:\teamsvoice.log -Append
            $params = @{
                Identity         = $adUser
                ObjectAttributes = @{edsvaTeamsVoiceEnabled = $false }
            }
            Set-QADUser @params
        }
    }
    finally {
        Disconnect-MicrosoftTeams | Out-Null
        Remove-Module MicrosoftTeams
    }
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
}

Parents

0 Stu.Pollock 5 months ago

Hi jody gilbert

Nothing is immediately jumping out as "the" problem, could we see a redacted version of the transcript?

It might be worth also adding a logging function, so you can output a message (including the time the function was called), so we can try and isolate where in the script it appears to be taking a long time/causing CPU spikes.
Cancel
Up 0 Down

Reply

Verify Answer

Cancel

0 jody gilbert 1 month ago in reply to Stu.Pollock

Apologies, just noticed I didn't respond to this. We fixed the issue last week by replacing the Get-QAD cmdlets with ADSI.

function Update-MSTeamsVoiceEnabledUsers() {
    try {
        $ProgressPreference = "SilentlyContinue"
        $ErrorActionPreference = "Stop"
        $appRegistrationCreds = 'CN=App Registration Details,CN=Library,CN=Script Modules,CN=Configuration'
        Invoke-Expression $(([adsi]"EDMS://$appRegistrationCreds").edsaScriptText[0])
        Import-Module MicrosoftTeams

        Connect-MicrosoftTeams @azureCreds | Out-Null

        [System.Collections.Generic.List[string]]$voiceEnabledUsers = 
            (Get-CSOnlineUser -Filter 'EnterpriseVoiceEnabled -eq $true -and LineUri -ne $null').UserPrincipalName

        $ldapVoiceDisabled = "(&(sAMAccountType=805306368)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(|(edsvaTeamsVoiceEnabled=false)(!(edsvaTeamsVoiceEnabled=*))))"
        $searcher = [adsisearcher]$ldapVoiceDisabled
        $searcher.SearchRoot = "EDMS://OU=Company Users,DC=somewhere,DC=local"
        $searcher.PropertiesToLoad.Add("userPrincipalName")
        $adVoiceDisabledUsers = @{}
        foreach ($result in $searcher.FindAll()){
            $adVoiceDisabledUsers.Add([string]$result.Properties.userprincipalname, $result.Path) | Out-Null
        }

        foreach ($user in $voiceEnabledUsers) {
            if ($adVoiceDisabledUsers[$user]) {
                # "Setting edsvaTeamsVoiceEnabled to true for $($user)" | Out-File C:\teamsvoice.log -Append
                $adsiUser = [adsi]$adVoiceDisabledUsers[$user]
                $adsiUser.Put('edsvaTeamsVoiceEnabled', $True)
                $adsiUser.SetInfo()
                # user found in the hashtable, removing to save time on future lookups
                $adVoiceDisabledUsers.Remove($user) | Out-Null
            }
        }

        $searcher = [adsisearcher]"(&(sAMAccountType=805306368)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(edsvaTeamsVoiceEnabled=true))"
        $searcher.SearchRoot = "EDMS://OU=Company Users,DC=somewhere,DC=local"
        $searcher.PropertiesToLoad.Add("userPrincipalName")
        $adVoiceEnabledUsers = @{}
        foreach ($result in $searcher.FindAll()) {
            $adVoiceEnabledUsers.Add([string]$result.Properties.userprincipalname, $result.Path) | Out-Null
        }

        foreach ($adUser in $adVoiceEnabledUsers.Keys) {
            if ($voiceEnabledUsers -contains $adUser) {
                # user found in the array, removing to save time on future lookups
                $voiceEnabledUsers.Remove($adUser) | Out-Null
            }
            else {
                $adsiUser = [adsi]$adVoiceEnabledUsers[$adUser]
                $adsiUser.Put('edsvaTeamsVoiceEnabled', $True)
                $adsiUser.SetInfo()
            }
        }
    }
    finally {
        Disconnect-MicrosoftTeams | Out-Null
        Remove-Module MicrosoftTeams
    }
}

Reply

0 jody gilbert 1 month ago in reply to Stu.Pollock

Apologies, just noticed I didn't respond to this. We fixed the issue last week by replacing the Get-QAD cmdlets with ADSI.

function Update-MSTeamsVoiceEnabledUsers() {
    try {
        $ProgressPreference = "SilentlyContinue"
        $ErrorActionPreference = "Stop"
        $appRegistrationCreds = 'CN=App Registration Details,CN=Library,CN=Script Modules,CN=Configuration'
        Invoke-Expression $(([adsi]"EDMS://$appRegistrationCreds").edsaScriptText[0])
        Import-Module MicrosoftTeams

        Connect-MicrosoftTeams @azureCreds | Out-Null

        [System.Collections.Generic.List[string]]$voiceEnabledUsers = 
            (Get-CSOnlineUser -Filter 'EnterpriseVoiceEnabled -eq $true -and LineUri -ne $null').UserPrincipalName

        $ldapVoiceDisabled = "(&(sAMAccountType=805306368)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(|(edsvaTeamsVoiceEnabled=false)(!(edsvaTeamsVoiceEnabled=*))))"
        $searcher = [adsisearcher]$ldapVoiceDisabled
        $searcher.SearchRoot = "EDMS://OU=Company Users,DC=somewhere,DC=local"
        $searcher.PropertiesToLoad.Add("userPrincipalName")
        $adVoiceDisabledUsers = @{}
        foreach ($result in $searcher.FindAll()){
            $adVoiceDisabledUsers.Add([string]$result.Properties.userprincipalname, $result.Path) | Out-Null
        }

        foreach ($user in $voiceEnabledUsers) {
            if ($adVoiceDisabledUsers[$user]) {
                # "Setting edsvaTeamsVoiceEnabled to true for $($user)" | Out-File C:\teamsvoice.log -Append
                $adsiUser = [adsi]$adVoiceDisabledUsers[$user]
                $adsiUser.Put('edsvaTeamsVoiceEnabled', $True)
                $adsiUser.SetInfo()
                # user found in the hashtable, removing to save time on future lookups
                $adVoiceDisabledUsers.Remove($user) | Out-Null
            }
        }

        $searcher = [adsisearcher]"(&(sAMAccountType=805306368)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(edsvaTeamsVoiceEnabled=true))"
        $searcher.SearchRoot = "EDMS://OU=Company Users,DC=somewhere,DC=local"
        $searcher.PropertiesToLoad.Add("userPrincipalName")
        $adVoiceEnabledUsers = @{}
        foreach ($result in $searcher.FindAll()) {
            $adVoiceEnabledUsers.Add([string]$result.Properties.userprincipalname, $result.Path) | Out-Null
        }

        foreach ($adUser in $adVoiceEnabledUsers.Keys) {
            if ($voiceEnabledUsers -contains $adUser) {
                # user found in the array, removing to save time on future lookups
                $voiceEnabledUsers.Remove($adUser) | Out-Null
            }
            else {
                $adsiUser = [adsi]$adVoiceEnabledUsers[$adUser]
                $adsiUser.Put('edsvaTeamsVoiceEnabled', $True)
                $adsiUser.SetInfo()
            }
        }
    }
    finally {
        Disconnect-MicrosoftTeams | Out-Null
        Remove-Module MicrosoftTeams
    }
}

Children

No Data

Endpoint Privilege Management

Troubleshooting a workflow running a script