SharePoint Insights – Keyset Does Not Exist Error

I had a scenario where SharePoint Insights was throwing a “Keyset does not exist” error in the ULS log. This prevented any audit entries from being uploaded to the Office 365 Compliance Center. The errors would look similar to this, and would be present on all servers in the farm.

The source of the error was Microsoft.Office.BigData.DataLoader.exe, which is the unfriendly name for the Windows Service ‘SharePoint Insights’. Working with Mike Lee, we determined the cause was that the Farm Admin account (which must be the account running SharePoint Insights), lacked permission to a particular MachineKey, though we didn’t know which particular key. Mike suggested running ProcMon for which I filtered for ACCESS DENIED on FIles only. The results looked similar to this:

At this point, we know exactly which file the Farm Admin is attempting to access. Fortunately the ULS tells us which certificate it is selecting prior to the ULS error. The certificate in the ULS is only identified by it’s thumbprint, so we still have to correlate the thumbprint with the certificate subject name and the above path from ProcMon. I made the assumption that it was a SharePoint-generated certificate based on the ULS errors so I started with the Local Machine SharePoint store. Using some PowerShell, it easy to identify the specific certificate.

The last property is key. The UniqueKeyContainerName is the same value as the file name in the Path from ProcMon. We identified that it was a Security Token Service certificate. Hybrid farms have two of these certificates; only one is in use so we need to verify we’re working with the correct certificate.

Unfortunately you cannot easily manage the Private Key ACL via the certificates MMC, so we’ll use PowerShell to manipulate the ACL instead. I’m putting the PowerShell below in two parts. The first part will get the certificate itself and enumerate the ACLs currently in place. The second block will set our new ACL, granting WSS_RESTRICTED_WPG_V4 with Read access to the certificate.

You can re-run the first block to confirm the ACL is now properly set. As soon as the ACL is set and the SharePoint Insight process runs again, you should no longer experience any errors during the upload process and begin to see audit content show up in the Office 365 Compliance Center.

This error is likely due to not running the Hybrid Configuration Wizard as the Farm Admin. In many environments, service accounts like the Farm Admin are not granted Local Administrator rights and many organizations use the best practice of denying the logon locally right SharePoint service accounts. That was the case in this particular environment, as well. Obviously it isn’t idea to have to run the above fix, but ultimately it is the only choice in many environments should they leverage SharePoint Insights.

Huge thanks to Mike Lee for working with me on this!

Leave a Reply