SPS 8.0 - Data cleanup failed

I have 2 SPS in configuration cluster.

Only the 2nd SPS is reporting this error after a successful backup:

SPS notification: Data cleanup failed

This is an SNMP alert generated on SPS host sps2 (master)
Cleanup failed.
Details:
Reason: unable to authenticate user [writer] for REST request [/_search]

SNMPv2-MIB::snmpTrapOID.0 SCB-SNMP-MIB::scbCleanupFailed
XCB-SNMP-MIB::description Cleanup failed.
XCB-SNMP-MIB::reason unable to authenticate user [writer] for REST request [/_search]

I don't find anything else in the log for scb or sys.
It only happens on 1 of the 2 SPS and not at all with my 2nd SPS test cluster.

Any idea why and what I can check?

I'm aware of the issue regarding the SPS backup since 8.0 mentioned in  Backup error; Could not create Elasticsearch snapshot 

Thanks

  • Hi,

    Please check in Policies > Audit Data Cleanup Policies
    To see if there are any misconfigured Cleanup policies with invalid Audit data query possibly?

    Thanks!

  • I only have the "vault_cleanup_auto_generated" with 360 days and origin: VAULT configured.
    Non of my sessions match with this policy.
    If I run all poilcies now I get the error but it also states "Audit data of 0 sessions will be deleted."

    I can delete this policy but I would like to understand the policy and the error first.

    Also if I create a new policy with 300 days and protocol: RDP or protocol: SSH it matches every session and when I run the policies it warns it would delete all sessions not just the older than 300 days. (I have 50 older session but it states it will delete 233)

  • This policy "vault_cleanup_auto_generated" is responsible for the cleanup of SPP related workflow data (fetched by the "SPP fetcher").

    As an example, the policy covers workflow data such as requests, approve / deny, return / revoke to an SPP asset, made by users / auditors.  

    If there is no active SPP fetcher role enabled in SPS side, then there is no data to be covered by this cleanup policy, however the policy will be still present on the SPS appliance, because this policy is auto generated.

    I would suggest raising a support service request and providing a support bundle after reproducing the issue for further investigation on the root cause of the mentioned error

    In regards to deleting more sessions that what appears in the Search interface, it may be possible there are orphaned audit trails in SPS node that are not visible in the Web UI possibly?

    Thanks!

  • Thank you. I opened a case and a reboot might have resolved the issue
    In regards to the deleting sessions: it seems like a misleading information. it states it will delete 180 sessions. but it only deleted older than 300 days. Running a second time it states it will delete 120 sessions. So around 60 were deleted and the information is kind of how many will be deleted in the future. 

    The Information in the popup is this:

    Run all policies now
    This deletes all audit data that matches the queries and is older than the retention periods specified in all polices.
    Audit data of 120 sessions will be deleted.

    I would expect it will delete 120 sessions right now, but it only wants to tell you the policies match 120 sessions which will be deleted in the future.