Interesting Exploits in Office 365 and SharePoint

Category Archives: File Shares

First Impressions – Microsoft Advanced Data Governance

Microsoft announced new capabilities for Advanced Data Governance at the Ignite conference in September 2016.  As of April 1, these features have been released into the O365 platform.  As with all O365 releases, these are rolling out in waves.  If you navigate to the Security and Compliance Center, you will see additional options under Classifications and Data Governance.

There are a significant number of features that have been released, but for the purposes of this discussion, we are going to focus on the specific capabilities surrounding information governance and policy.

Retention Policies and Classification Labels

Microsoft’s capabilities around retention are not new.  Both Exchange and SharePoint have featured mechanisms to define how long content items should exist.  Many of these manifesting themselves over 10 years ago.  What is new with the recent announcement and update is Microsoft’s unification of retention policies across the Office 365 (O365) service.  Within a series of mouse clicks, one can create a retention policy and apply it to Exchange (email and public folders), SharePoint sites, OneDrive for Business, Office 365 Groups and Skype for Business content.  To an organization, this now provides the ability to create and enforce a number of information governance initiatives. 

Layered on top of Retention Policies are Classification Labels.  Classification Labels allow organizations to apply a specific information governance rule to content manually.  The Classification Label sets can be published to different locations across the Office 365 platform.  The result provides users the ability to easily follow information governance policy without having to have deep knowledge or training.  Apply a Classification Label of “External Proposal” to a new sales proposal and the user is done.

Alternatively, Classification Labels can be defined and applied automatically and/or by default to a set of locations.  Maybe your organization has a mergers and acquisitions group.  A Classification Label can be defined and automatically applied to all content created in that group regardless of where it is stored.  The added benefit of a Classification Label over a Retention Policy is that the label will actually manifest on the content, whether a document or email.

Retention Policies and Classification Labels offer the ability to ubiquitously apply to all content in the platform as a global policy.  Alternatively, they can be applied to very specific content, in specific targeted locations.  This flexibility offers organizations great power to solve many diverse and unique information governance challenges.

Before organizations jump into this pool with both feet, I suggest that they take a deeper look at these capabilities so that they obtain the results that they expect.

Retention and Records

The new Advanced Data Governance (ADG) features provide the ability to perform content retention as well as record declaration.  Retention, as implemented in the ADG context retains content for the specified period of time.  This doesn’t mean that the content is inaccessible.  Users may continue to work with the content including editing, sharing and collaborating.  Content that is under a retention policy behaves like any other content to the user.  What is different is when a content item is deleted.  The ADG features manage the preservation of the content so that it is not lost.

Retained content is handled differently than content marked as a “record”.  This is a unique option for a Classification Label and removes the ability to modify the content in any way.  This prevents users from modifying the specific item and will change the behavior experienced during the normal course of working.

Deletion and Destruction

The process that occurs when content is no longer retained may be defined as a delete action.  The retention policy can be defined to do nothing at the end of retention, or perform a delete.  It is important to note that delete in this case doesn’t mean “delete immediately”.  ADG uses the recycle bin to provide a staged deletion process.  Therefore, “deleted” content may still exist in the recycle bin for a period of time.

In the future, there are plans to put approval process in place for the retention policy that will provide organizations the ability to create more formal processes around information destruction. 

Date Calculations

With this release, ADG provides date options for the calculation of the retention period.  Standard dates such as Created and Modified are available and an option for when the content was labeled (in the case of Classification Labels).

Currently there is no option to support custom date fields or event-driven (also known as retention triggers) retention.  These are going to be available soon per Microsoft.


After being involved with the preview as a Microsoft ISV and working with several joint partners we have some initial impressions of these new features.

  • Microsoft has finally moved to unification of policy management – this is a big plus because we no longer manage separate, application specific policies
  • Application of Retention Policies and Classification Labels – definition of the policies and labels is quite easy and operates through a very modern wizard, stepping even the novice through the process.
  • Timeliness of Processes – in our experience, the policy application service level agreements aren’t immediate, some taking as long as seven days.  We are sure that as ADG gets wider adoption and Microsoft gains experience with the service, these will come down.
  • Not your traditional records management – there is some adjustment that will be needed for more traditional organizations using legacy records management solution to adapt to these new concepts.  For some organizations, these capabilities will augment more regimented processes to address the ever-increasing proliferation of content and the risks it represents.
  • Terminology – we can only suggest taking your time and understand what each of these new features is doing.  We found ourselves a little confused more than once with the use of terminology.
  • Enterprise Perspective – one of the biggest challenges we faced was understanding the what, when and where of the Retention Policies and Classification Labels.  Since each of these is a simple listing of defined policies and labels, it becomes very difficult to track where certain policies are applied and what labels are available.   We suggest starting with very broad definitions and picking relevant naming conventions (you can name and describe these any way you want).  Microsoft is gathering feedback in this area, so we hope to see good things coming.
  • What about other sources – as stated previously, Microsoft has made a great stride with these capabilities.  If your organization has other locations or applications that need to have policy assigned, you will need to look for solutions that extend the platform.

In conclusion, Microsoft deserves credit for listening to its clients and taking this leap forward.  For those of us committed to the platform, we are happy to see these capabilities.  While adding some great capabilities, we believe that organizations will find that there will be compliance requirements and information handling processes that will require broader and deeper functionality in specific areas.


Pre-Processing Content for Migrating to SharePoint 2010

I have recently been working with a number of customers to establish information governance in their SharePoint 2010 environments.  The question always comes up is “how do we get content off the network shares into the SharePoint environment?”

There are a number of commercial tools that will analyze, cleanup and migrate network share content to SharePoint.  These tools come from vendors such as StoredIQ, Active Navigation, EMC Kazeon and Metalogix.

If you are going to be migrating file shares to SharePoint, regardless of the tool being used, there is a basic process that I think you should be taking. This process is broad, but could help you get started thinking about your environment. You can see the basic steps below.

You may look at that process and be thinking that it doesn’t really say anything, nor does it really help you.

As I said, there are many tools out on the market today to help in this endeavor.  These tools can help you discover all the files you are looking to migrate; they can also help you filter out files by date, specific strings, or other information that you may not want to migrate to SharePoint.

However! There are a couple tools that you can use right now that can help. They are PowerShell and Microsoft Excel! Yes, PowerShell and Excel!

Take the example of a network file share you want to migrate into SharePoint. The share is N:\team1. In the process above, the first step is to discover what files are in the file share, how old they are, what types there are etc. Of course, you could go through and manually look at them one by one. Or, you could ask the person responsible for the file share to do that.  Or you could use PowerShell to iterate through and output the results to a CSV file. Simply by running this command:

Get-Childitem –recurse \\Ndriveserver\team1 |Export-CSV c:\export.csv

This process will create a CSV file that can be opened with Excel as you can see below.

With the power of excel sorting and filtering you can scroll through and really look at a ton of data about the files. Such as Extension, Last Access Time, Last Write Time etc.

This could be a great help; however, it could also be too much. Say you only want to see all the files with exe, MSI, and MP3 extensions? You could use the same command but use the Include parameter.

Get-Childitem –recurse \\Ndriveserver\team1 -include *.exe, *.MSI, *.MP3 |Export-CSV C:\export.csv

Or, if you want to narrow down your list by excluding files you don’t care about like DOC, XLS, and PPT since these are probably more natural files to migrate.

Get-Childitem –recurse \\Ndriveserver\team1 -exclude *.doc, *.xls, *.ppt |Export-CSV C:\export.csv

At this point, congratulate yourself, you have discovery, now you can run these PowerShell commands and use Excel to sort and filter.

Now, let’s go to the next, Cleanse.  We want to remove what is often termed ROT (Redundant, Obsolete and Trash or Trivial) files. Once again there are many products that make this part of the migration or import process.

However, you can do some cleaning with PowerShell as well. When you come up with a list of files you want to get rid of such as all the MSI files in the \\Ndriveserver\team1\Marketing directory, you can use the Remove-Item to get rid of them.

Remove-item \\Ndriveserver\team -recurse1\Marketing\*.pptx

You could also do something a little more fun.

get-childitem \\Ndriveserver\team1\Marketing -include *.MSI -recurse | foreach ($_) {remove-item $_.fullname}

However remember that you want to test your process for jumping in with both feet. You can use a parameter on the PowerShell command to test before you actually remove.

get-childitem \\Ndriveserver\team1\Marketing -include *.MSI -recurse | foreach ($_) {remove-item $_.fullname -whatif}

The –Whatif command will output what the command would have done, but not do it.

Like the shot below if I was to delete all the PPTX files

Running the command with the –Whatif parameter will help you make sure you are aware of exactly what the command would delete.

So now we can identify file types, and that may be useful in the case of log files or obscure application output files; however that typically is not all the criteria that you would practically use. What if you want to enforce a policy of removing OLD files? For example, what if you wanted to remove anything that has not been modified in the past year? With the capabilities of PowerShell you can do that!

Get-Childitem N:\Marketing -recurse |Where-Object {$_.LastWriteTime.Date -le (Get-Date).Date.AddDays(-365)}|foreach {remove-item $_.fullname }

Finally, you do have another option that can take advantage of that CSV file you are creating. Once you create that export file (CSV) you can delete all the entries of files you wish to keep, thus leaving in all the ones you want to delete. Then use the Import-csv PowerShell command and use that as the input to the remove-item command as follows:

Import-CSV c:\export.csv | foreach {remove-item $_.fullname }

The result is any file listed in that CSV goes away. This is one where you definitely want to be careful and ensure that you are only listing out files you really want to get rid of!  Remember the WhatIf!

Hopefully, I’ve given you some insights how to Discover and Cleanse your network file shares in preparation of migrating them to your SharePoint environment!