Azure Blob Storage for SharePoint Documents

[This post is based on the SharePoint 2016 Preview — Functionality is not guaranteed to be supported, configured identically, or exist in RTM. This functionality is highly likely to not exist in the RTM product and is shown here as an academic exercise.]

SharePoint 2016 includes some very interesting features that allow us to take a look at what is being used within SharePoint Online (“the service”). One of these features is Fort Knox, which is per-file encryption blobs residing in Azure, instead of SharePoint’s Content Database. Leveraging Azure Blob Storage for SharePoint documents is not a straight forward process, and is currently not supported by the SharePoint Product Group, but below are the steps to follow in order to enable it for demonstration purposes. If support for this is interesting for you or your customers, make sure to let the SharePoint Product Group know via UserVoice or an RFC through standard Microsoft support offerings. If this process was ever supported, the assumption would be that it will only be supported for SharePoint farms running in Azure Virtual Machines and where the farm is in the same Azure region as the primary Azure Blob Storage account.

To set this up, you’ll need to have an Azure Subscription, and an Azure Active Directory user account (this user account does not need any assigned license within Office 365, if your Azure subscription is tied to Office 365).

Create two Azure Blob Storage accounts. These accounts have the following requirements:

1) Must start with the letters “spo”.

2) Must end with the letter “c”.

3) Cannot be longer than 10 characters total, including “spo”, and “c”. In my examples, I used spo1splabc and spo2splabc.

4) Must have two Azure Blob Storage accounts. This will not work without two accounts!

One Storage will be used as the Primary, the other as a Secondary (or “DR”). Give the Azure AD account Owner rights over the Azure Blob Storage accounts. The container may stay Private R/W.

On your SharePoint server(s), install Microsoft.WindowsAzure.Storage.dll. You can use GACUtil to install the binary to the GAC. The last pre-req is to set the SharePoint Farm Credentials to connect to Azure.

You must create Shared Access Signature (SAS) URIs. Using the GenerateSas project, you will modify the app.config included with the project with the Connection String from your Azure Blob Storage account. This key will look like:

You must generate a unique SAS URI per Blob Storage account. Note that if you want to change the container name used (as this will be used elsewhere during this guide), adjust this line, changing “sascontainer” to your desired value. The container value must be a minimum of 3 characters. This guide will continue using “sascontainer” as the value, since it is the default included with the project.

In addition, there is another change to this project, which sets the expiration value at the maximum and grants further permissions.

Take the output of the console application and place it into a PowerShell variable, one for each container. For example:

Next, grab the SharePoint Farm Id.

This next cmdlet is lengthy. You’ll also note I’m not strictly following the purpose of each parameter, instead using the same key for multiple purposes.

This adds the keys to the SharePoint server’s registry. Provide the WSS_WPG account with Read/Write permissions to the following key, which needs to be done on each SharePoint server.

Run the following PowerShell cmdlet:

Create the following file on each server, which needs to have the same content on each server.

The content of this file needs to be, at a minimum, this:

The next steps are to modify the database directly via T-SQL. You must set these values correctly for the Azure Blob Storage upload to take place.

The first allows for SharePoint to directly write to ABS while the second allows the ABS Drain Timer Job to run (drain meaning “move all content to ABS”). In addition, you must provide where to migrate the blobs to along with where to store the blobs!

The format for  _AbsLocationInfo and _MigrationTargetAbsLocationInfo is  NearABSFQDN;/NearContainerName;FarABSFQDN;/FarContainerName;BlobExpirationTime . To add all of these values to the SharePoint Content Database, use the following T-SQL, where the variable is, for example, “_EnableDirectWriteToAbs” and the value is “true”.

Finally, you’re ready to migrate to Azure Blob Storage! Run the job-drain-docstreams-abs timer job:

This will execute the job, and if everything was set up correctly, encrypt and migrate your data from the DocStreams table to Azure Blob Storage. The result will be similar to this:

ABSDocStreamsTable

The ABSId value corresponds to the file name of the Blob, as a GUID  while ABSInfo is information about the ABS Container, version, and encryption information. Here we can see the corresponding ABSId, 0EB745E7-… in the Azure Blob Storage container:

ABSBlobStoreDoc

 

As new documents are uploaded to the Content Database, files will be written to ABS instead of the Content Database.

If for some reason the migration from the Content Database to Azure Blob Storage fails and you want to retry after resolving any issues, you must perform two steps:

 

7 Comments

  1. That means also each individual document has its own url and can be accessed via REST directly + be made accessible and cached via Azure CDN. Storage limitations in traditional content databases will be history too! :-)

  2. Very interesting article and interesting to see whether and how this will be supported.

    How did you figure out how to do everything you just described? Stuff like setting read/write permission on the registry key etc.?

    • Dennis, it is _very_ unlikely that this will be supported. That said, if this is of interest to you, I would suggest opening up a discussion on https://sharepoint.uservoice.com to let the Product Group know that you would find this feature useful. As for finding out how to do this, it was done via .NET Reflector and Process Monitor.

      • You really did this all by reflecting and procmon… crazy :-)

        I think it would not only be of interest to me. Azure Blob storage is suited a lot better to store files than SQL server itself is. This would make some nice RBS scenarios possible. But yeah, I also deem it unlikely to become a supported scenario.

  3. Pingback: SharePoint 2016 Technical Preview vu depuis PowerShell | Le Post de MCNEXT

  4. Pingback: SharePoint & Office 365: Recopilatorio de enlaces interesantes (II)! - Blog de Juan Carlos González en Geeks.MS

Leave a Reply