ARS PowerShell script to "Move" users from one group to another.

I have a .csv with 2 columns. Column 1 is called SourceGroup with a list of groups, and the other is DestinationGroup with a different list. I have a script I am trying to use that will Add the users from SourceGroup to the DestinationGroup, after the user will be removed from the source group. The error I am receiving is that they are already a member of the DestinationGroup. The issue is they are an indirect member, and I want to make them active members. Is there something I can add to my script to have it not check for indirect memberships? Here is the script I am using. I was hoping to not need to use the AD native tool as that would require me escalate up to Domain admin.

 clear
$admin = read-host "Enter username(domain\username)"
$pw = read-host "Enter password" -AsSecureString

$LogFile = "H:\ticketstuff\success.txt"
$LogFile2 = "H:\ticketstuff\failed.txt"

connect-qadService -ConnectionAccount $admin -ConnectionPassword $pw

$list = import-csv “H:\ticketstuff\listtest.csv"


foreach( $Group in $List ){
    $Members = Get-qadGroupMember $Group.SourceGroup | select UserPrincipalName
    $Member = $Members.UserPrincipalName 
    $S = Get-QADGroup $Group.SourceGroup | Select-Object -ExpandProperty name
    $D = Get-QADGroup $Group.DestinationGroup | Select-Object -ExpandProperty name
    foreach ($User in $Member){
    Try
        {
        add-qadgroupmember -identity $Group.DestinationGroup -member $User | Where{$_.objectClass -eq 'User'} -ErrorAction Stop
        "User $User added to group $S" | Add-Content -Path $LogFile
        remove-qadgroupmember -identity $Group.SourceGroup -member $User | Where{$_.objectClass -eq 'User'}
        "User $User removed from group $D" | Add-Content -Path $LogFile
        }
    catch
          {
	        "Error $User already in group $D" | Add-Content -Path $LogFile2
          }
     }
}
  
Disconnect-QADService