About SharePoint OnPrem Multitenancy

As per https://docs.microsoft.com/en-us/sharepoint/what-s-new/what-s-deprecated-or-removed-from-sharepoint-server-2019 it is clear that multi tenancy is supported in SPS 2016 and deprecated with 2019. But…

On-prem multi-tenancy is a configuration that is extremely difficult to set up and maintain.  We’ve been strongly discouraging customers from using this configuration unless they already deployed it during 2010/2013.  And if they have, then upgrading to a new version of SharePoint should be looked at as an opportunity to get off this configuration.  We only offer “limited support” for this configurations both SPS2013 and SPS2016 because the out-of-box experience is incomplete, so customers end up having to write a lot of custom code to create a good end-to-end experience.



Site Retention Policies keep sending notification emails to end users even postponed sites

Problem definition:

You are using Site Retention Policies in standard SharePoint 2013/2016 teamsites. The feature works as expected although if site owners extend their teamsite, the site keeps sending notifications about “the site is about to expire and will be deleted”. As it displays already the new deletion date which is one year ahead.

Well, out of the box SharePoint behaviour has been designed that any postpone should be short term and that postpone duration SharePoint keep notify you, even if the site has been postponed.But there is a glich in this design, the site owners can postpone a site over years. Of course in that duration every week (Actually whenever the “Expiration” timer job runs) if you get notification, it will be very annoying.

Luckly we have some workaround to mitigate this.

Before going to workaround, I would like to give some information

How to configure Site Policies, please read following article.
“Site Settings -> Site Policies”


– “Site Policy” Feature: Site Settings -> “Site Collection Features” -> “Site Policy”. It is the feature you should already enabled to able to use “Site Policies”.

– “Expiration Policy” Timer job: Each web application we have one “Expiration” timer job. That timer job has responsible to expire operations.
Enumerates list items and looks for those with an expiration date that has already occurred. For those items, runs disposition processing. Disposition processing most often results in deleting items. But it can perform other actions, such as processing disposition workflows.
This timer job’s default schedule is weekly. So you would expect notification emails fires weekly.

-For every SPWeb object ( Root or subwebs) we have; Site Settings -> “Site Closure and Deletion” settings.We have assigning related policies here for a specific SPWeb Object (even root site web object)

– “Project Policy Item List” . This is the hidden list that your policy related items/configurations are stored here.When you assign a policy for a site, it stores here. Every site collection have one of this list instance if you enabled the “Site Policy” Feature. So you can find this list in Site Collection’s root web. (Not under sub webs).

-Also we have several policy system EventRecievers. It is important because we should disable them if we going to play around with policy internal settings.

Let’s give an example.
Assume we have created a new site policy:

“Deletion Event”: Site Created Data + 1 Year .
“Send an email notifications to site owners this far in advance of deletion:” -> 3 Months
“Send follow-up notifications every:” -> 7 Days
“Owners can postpone imminent deletion for”  1 years .

(The below picture say 14 day, consider it 7days pls)

Current date of the server : 05/01/2017

So what this information tell us:
If we assume we have created a site 05/01/2017 , this site will be deleted on 05/01/2018
Before deletion of 3 months , we start to get notifications. According to our example starting by 05/10/2017. (You will get first notification exactly when the Expiration timerjobs run on that week)

So i have created a brand new subsite and assigned this policy to that subweb from “Site Closure and Deletion”

If we look in “Project Policy List Item” table. We learn more information. You can use below powershell script to get information about related item.
(You need to adjust the script for finding correct list item id, I will not do it here)

If ((Get-PSSnapIn -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null )
{ Add-PSSnapIn -Name Microsoft.SharePoint.PowerShell }

$url = “<your site collection url>”
$rname = “Project Policy Item List”
$site = Get-SPSite $url
$rootweb = $site.OpenWeb()
$rList = $rootweb.Lists[$rname]
$item = $rList.Items[“<please locate the correct item id>”]
write-host “ExpireDate           :” $item[“_dlc_ExpireDate”]
write-host “ProjectExpirationDate:” $item[“ProjectExpirationDate”]
write-host “ProjectCreateDate    :” $item[“ProjectCreateDate”]
write-host “LastRun   :” $item.Properties[“_dlc_LastRun”]
write-host “PROPERTIES”
write-host “XML”
$item.xml.Replace(“ows_”,[System.Environment]::NewLine + “ows_”)

The result is

ExpireDate           : 05/10/2017 2:27:27 PM  +9 Months.
-This is the value of date, next time “Expiration” timer job notice that should do something about it.
-It is not a real expiration date, it is a changable varible which “Expiration” timer job can calculate and change in time to time.
ProjectExpirationDate: 05/01/2018 2:27:27 PM  +1year.
-This value means when we delete the site according to our formula.
ProjectCreateDate    : 05/01/2017 2:27:27 PM  +0 time.
-This value when we have created the item policy. It is the beginning reference time.

So basically when we pass the date “05/10/2017”, Depends on “Expiration” timer job schedule in this week of the date, the timer job will send the email notification about our site will be deleted in 3 months, Lets assume timer job run on 07/10/2017 (2 days after “ExpireDate” value) your site owners will get first notification about site deletion. Afterwards timer job will update “ExpireDate” value by adding 1 week = 14/10/2017 (based on setting we defined “Send follow-up notifications every”). Also will update/add some other properties like “_dlc_LastRun” it will be 07/10/2017) .

On date 14/10/2017 is the next time “Expiration” timer jobs runs, it will send the second notification about site deletion.Afterwards timer job will update this date adding another 1 week. 21/10/2017. This goes until we reach “ProjectExpirationDate” on that date the object will be deleted. (or closed, depend on how you configure the policy)
Ok. So lets have a look some other important parameters.
ItemRetentionFormula -> It shows the formula of when we start first do something about this record -> For the first time,  “ExpireDate” calculated with this formula.But “ExpireDate” property will be changed afterwards when that date has come by “Expiration” timer job.

PS C:\Users\Administrator.CONTOSO> $item.Properties[“ItemRetentionFormula”]
<formula id=”Microsoft.Office.RecordsManagement.PolicyFeatures.Expiration.Formula.BuiltIn”><number>-3</number><property>ProjectExpirationDate</property><propertyId></propertyId><period>month

Some other important properties.
_dlc_policyId                  0x010085EC78BE64F9478AAE3ED069093B996300ACCF30C2E8DFDE4CB3D2D69F6C58E43C
ows_ProjectWebUrl=’http://contososp:9090/sites/corpa/SharePointHub, /sites/corpa/SharePointHub’ -This is my subsite name

ows_ProjectCreateDate  -> We already explained this above
ows_ProjectExpirationDate -> We already explained this above
ows_ProjectIsClosed=’0′ -> If you selected the option close the site before deletion (meaning make it not reacable by users)
ows_ProjectNumberOfPostpone=’0′ -> It will show the number that you can understand any postpone happens.
ws__dlc_ExpireDate-> We already explained this above
ows_ContentType=’MyDeletePolicy’ -> this is same as the Policy name.Well the system works via content type structures behind the scene.

Lets return our problem. The problem begins when your site owner decide and postponed the deletion afterwards he/she got the first notification. Well according to our settings it will be postponed for 1 year ahead. But the problem the owners will continue to get emails for every week (we set “Send follow-up notifications every” it 7 days). I said 🙂 it is annoying.

What happens after you postpone in related item properties in “Project Policy List Item”
ows_ProjectNumberOfPostpone=’1′ -> changed “0” -> “1”
ows_ProjectCreateDate=’05/01/2017 2:27:27′  -> It is not changed.
ows_ProjectExpirationDate=’05/01/2019 2:27:27′ -> but this date increased as 1 year now in 2019 !.
We have new property named


There is only one workaround, can only applicable with Powershell. if we delete _dlc_LastRun and newly added property _dlc_ItemStageId and run “Expiration” timer job afterwards. Timer job will do a recalculation and correction. It will going to apply again the first time ItemRetentionFormula but this time the ProjectExpirationDate is in 2019.(Remember, it was updated when the site owner postponed). So correction will reset the “ExpireDate” value and you will not get anymore notification emails until  “05/01/2019 minus 3 months”, All Good 🙂

Here is the powershell to fix that issue:

If ((Get-PSSnapIn -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null )
{ Add-PSSnapIn -Name Microsoft.SharePoint.PowerShell }

#Site Collection Level.
$site =get-spsite <site collection url>

# Open the RootWeb
$web = $site.OpenWeb()

#gather  Project Policy Item List hidden list
$list = $web.Lists[“Project Policy Item List”]

#There will be several subsites or different policy item in the list based on usage.
#Need to locate correct policy item in the list with related site or subsite.
#Print all items in that list to find out the relate (site or subsite object)
#Please add your logic based on your requirements.For example you can use ProjectWebGuid or ProjectWebUrl for filter out your related item.I will leave it to you.

#gather the item.
$item = $list.Items[<related item id>]

#-Update procedure for the ExpireDate
$assembly = [Reflection.Assembly]::LoadWithPartialName(“Microsoft.SharePoint”);
$type = $assembly.GetType(“Microsoft.SharePoint.SPEventManager”);
$prop = $type.GetProperty([string]”EventFiringDisabled”,[System.Reflection.BindingFlags] ([System.Reflection.BindingFlags]::NonPublic -bor [System.Reflection.BindingFlags]::Static));
$prop.SetValue($null, $true, $null);


$prop.SetValue($null, $false, $null);

+Then run “Expiration” Timer job for related web application.

I strongly suggest, please do some test and practice the script on your test environment before appling it to production!
If you do something incorrect, you can easly messed up your policy items.Or you may call Microsoft Support to help.

About SharePoint with 16+ Cores

Well, If you update .Net 4.7.2 or higer it is OK, otherwise it is a bad idea.

More speficially,
ReaderWriterLockSlim with reentrant have design limitation which can lead serious performance down for previous .Net versions. And Sharepoint pretty much depended on this thread syncronization object. Specially Blob Cache and ObjectCache wraps and using them. More CPU causes more thread contention, excesive locking thats brings slowness.

Example callstacks:
SPReaderWriterLock named [BlobCache] waited 43992 milliseconds to acquire lock. Call stack:
at Microsoft.Office.Server.Utilities.SPReaderWriterLock.AcquireLock(Boolean readerLock, Boolean upgradable, Boolean throwException)
at Microsoft.Office.Server.Utilities.SPReaderWriterLock.AcquireLock(Boolean readerLock, Boolean upgradable)

Microsoft_Office_Server!Microsoft.Office.Server.Utilities.SPReaderWriterLock.AcquireLock(Boolean, Boolean)
Microsoft_Office_Server!Microsoft.Office.Server.ObjectCache.SPCache+MossObjectCache.UpdateUsageMap(System.String, UInt32, UInt32)
Here are some threads about the problem.

These issues has been fixed with .Net Framework 4.7.2 version!.

Pls check .NET Framework 4.7.2 Release Notes

Again don’t think about 64cpus makes more performance. It depends of the software boundaries/limitations and several other things..
My suggestion, scale up with multiple machines. It is more cheaper and stable. ! Not with excessive hardwares. (8 cores are fine 🙂 )

If you have that monster machines, don’t worry you can  use it with HyperV and scale up VMs.

After updating SharePoint 2013 to November 2017 CU or later you may not be able to open documents with Office


This issue mostly happens if you update your sharepoint from command-line by using psconfig.exe and when you miss the correct parameters.

PSConfig.exe -cmd upgrade -inplace b2b -wait -cmd applicationcontent -install -cmd installfeatures -cmd secureresources -cmd services -install

Thanks to Rodney for excellent work to detecting the issue .we have an easy workaround of this.But we don’t like much to copy/paste dlls around.

Instead of manullay copy/paste the stssoap.dll around bin folders and if you already run psconfig.exe by missing applicationcontent -install parameters , you can use following powershell commandlet ;

for more information about PSCONFIGUI.EXE and PSCONFIG.EXE please read outstanding article by my colleague Stefan Gossner

Mainstream support for SharePoint 2013 will end in 6 months

Mainstream support for SharePoint 2013 will end on April 10th, 2018:

After this date only security fixes will be provided for SharePoint 2013. Regular hotfixes can no longer be requested.

If not already done we recommend to start planning the migration to SharePoint Server 2016 as soon as possible.

Unable to open BDC Service Application UI from Central Admin site

Here is the issue definition that If we go to Central Admin – Manage service Applications -> Businees Datas Conectivity Service Application we obtain an error:

“Something went wrong” and a Correlation ID
Error message seen:
Event ID 8085 Event Viewer The BDC Service application Business Data Connectivity Service is not accessible. The full exception text is: Access is denied.
At logs:
SPIisWebServiceAuthorizationManager: SPIisWebServiceApplication with name ‘Business Data Connectivity Service’ and type ‘Microsoft.SharePoint.BusinessData.SharedService.BdcServiceApplication’ received request with ServiceSecurityContext whose primary identity has no valid data to check against ACL.
An exception occurred while writing a service call usage entry. Exception details: System.ObjectDisposedException: Safe handle has been closed
at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
at Microsoft.Win32.Win32Native.GetTokenInformation(SafeTokenHandle TokenHandle, UInt32 TokenInformationClass, SafeLocalAllocHandle TokenInformation, UInt32 TokenInformationLength, UInt32& ReturnLength)
at System.Security.Principal.WindowsIdentity.GetTokenInformation(SafeTokenHandle tokenHandle, TokenInformationClass tokenInformationClass)
at System.Security.Principal.WindowsIdentity.get_User()
at System.Security.Principal.WindowsIdentity.GetName()
at System.Security.Principal.WindowsIdentity.get_Name()
at Microsoft.SharePoint.Utilities.SPUtility.GetCurrentThreadUserLogin(Boolean fFallbackToEnv)
at Microsoft.SharePoint.Administration.SPUsageManager.LogUsage(SPUsageEntry usageEntry)

The BDC Service application Business Data Connectivity Service is not accessible. The full exception text is: Access is denied.

From Central Administration Site when we try to open BDC service we have making a WCF request to Business Connectivity Service

Name=Request (GET:http://contoso.com:3760/_admin/BDC/ViewBDCApplication.aspx?AppId=ec61c2eb-a874-4dfd-8245-0476da3d2731)
WcfSendRequest: RemoteAddress: ‘http://contoso.com:32843/b02ca86c7cb94143bb8277579dbc505c/BdcService.svc/http&#8217; Channel: ‘Microsoft.SharePoint.BusinessData.SharedService.IBdcServiceApplication’ Action: ‘http://www.microsoft.com/Office/2009/BusinessDataCatalog/BusinessDataCatalogSharedService/MetadataObjectCreate&#8217;
WcfReceiveRequest: LocalAddress: ‘http://contoso.com:32843/b02ca86c7cb94143bb8277579dbc505c/bdcservice.svc/http&#8217; Channel: ‘System.ServiceModel.Channels.ServiceChannel’ Action: ‘http://www.microsoft.com/Office/2009/BusinessDataCatalog/BusinessDataCatalogSharedService/MetadataObjectCreate&#8217;

We have facing an authentication problem on Claims authentication. Looks that “User is not authenticated”

So it bring us to “Security Token Service” Application before calling BDC request

Claims Authentication af3y2 VerboseEx STS Call Claims Windows: Adding claim with type ‘http://sharepoint.microsoft.com/claims/2009/08/isauthenticated&#8217;, value ‘False’, value type ‘http://www.w3.org/2001/XMLSchema#string&#8217;, issuer ‘SharePoint’ and original issuer ‘SecurityTokenService’.
Claims Authentication af3y1 VerboseEx We are copying claim with type ‘http://sharepoint.microsoft.com/claims/2009/08/isauthenticated&#8217;, value ‘False’, value type ‘http://www.w3.org/2001/XMLSchema#string&#8217;, issuer ‘SharePoint’ and original issuer ‘SecurityTokenService’.

For Resolution and TroubleShooting suggestions

-> Check BDC Service Application has only Anonymous Authentication has enabled and “windows authentication” has disabled.
-> Check The Security Token Service Authentications are “Anonymous” and “Windows Authentication” has enabled.
-> Check IIS > SharePoint Web Services > Only Windows Auth should be selected.
-> Check BDC Service Application Anonymous Authentication Identity has set for “IUSR”
-> Check Top Level IIS Anonymous Authentication Identity has set for “IUSR”

1. Open IIS manager
2. Highlighted server name
3. Select Authentication from center pane
4. Highlight “Anonymous Authentication” and be sure it is Enabled
5. Click on “Edit…”
6. Select the “Specific User” radio box and click “Set”
7. Enter IUSR in the “User name:” box on the Set Credentials window.
— Note you do not need to enter a password.
8. Click OK to apply, then OK to apply.

Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

*** UPDATE November 2018, SCOM team has released a fix for SCOM 2016 (UR6 for SCOM 2016) and mitigated the APM issue. (If you are using SCOM 2012 unfortunately this is still not supported) Personally i do not recommend using any kind of profiler with a SharePoint production enviroment. 

More info:

** UPDATE June 2018. Recently we had another discussion with our product group about disabling LoaderOptimization. There are some rare scenarios that we have no other option that disabling LoaderOptimization as workaround. Not for APM scenarios, APM is still not supported.

The error when the issue happens:
Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

Recently we have facing many issues with above exception .Effected products are SharePoint 2013 and SharePoint 2016.Mostly the cases are open as CritSit because of SharePoint down.

• We typically see these errors with non-SharePoint code running, such as SCOM’s APM agent or 3rd party monitoring software, or very unlikely .Net patches.
• Specifically in our case, we found the environment running APM monitoring Agent: APMAGT
• The most likely that the CLR was trying to optimize a reload of the instrumentation agent with an incompatible profiler loading sequence.

If there is a profiler exists, .Net Profiler api , gives a chance for a tool to examine and modify the IL before it gets to the JIT.  This allows one to add probe calls for things like code coverage, performance, or so whatever. As you imagine manuplating IL by profiler that may cause several abnormalities.

Cascade of events leading to the exception when executing the .Net Profiler callback though is in native code, instantiates managed code. All the instances in the process should be for the FullTrust Application Pool AppDomain, however since a generated call if it is not mapped to an AppDomain,(Yes it is possible, simple example IIS application pool ping feature)  it loads it on Default AppDomain which is not in FullTrust it causes a problem of loading an assembly with different permission set.

“Today only the top level assembly’s grant set is checked. If later CLR sees an incompatible security grant set for any assemblies in the binding closure, CLR will not be able to recover from the error and will fail to load the assembly in the new appdomain. Essentially the host is responsible for not violating this constraint.”

I have seen many misredirected articles even in technet blogs and incorrect resolutions are suggested for resolving this problem.

One of them changing legacy cas model settings.
Which is out of the box for SharePoint products the Legacy CAS model is a requirement.So we have this value in every SharePoint web sites.
trust level="Full" originUrl="" legacyCasModel="true"
(This is correct!!!)
People are resolving this problem by setting legacyCasModel=”false” which is NOT SUPPORTED .

You shouldn’t change it because of backward compatibility and dependancy reasons we need that. Disabling this property cause anomalies.

Please be aware meaning of NOT SUPPORTED ,does not mean your SharePoint not work. It may work but  we may not give support unless you rollback your changes or can not forseen any other anomalies .

Another incorrect resolution.
Create a new registry DWORD value called LoaderOptimization and give it the value 1 within the key
Perform IISRESET and check Web Apps behavior.

In key ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework’, create a new ‘DWORD (32-bit) Value’ named “LoaderOptimization” with a value 1 (either in decimal or hexadecimal as they are the same).
In key ‘HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework‘, create a new ‘DWORD (32-bit) Value’ named “LoaderOptimization” with a value 1 (either in decimal or hexadecimal as they are the same).

This configuration will disable the loader optimization of assemblies that provoke the aforementioned behaviour by setting assembly loading into SingleDomain mode.

And this is NOT RECOMMENDED** for SharePoint by Microsoft.

First of all, ASP.NET is using  by default LoaderOptimization attribute as MultiDomainHost. (System.Web/Hosting/ApplicationManager.cs).  It is not specific to SharePoint.
There are some reason for why we don’t recommend this:
1) These settings are not tested by product group. (It is not design to work in SingleDomain mode)

2) Let’s have look what is LoaderOptimization enumeration
We have setting “LoaderOptimization” as 1 which means the “SingleDomain” mode.
“Indicates that the application will probably have a single domain, and loader must not share internal resources across application domains.”

SharePoint have many assemblies with many different locations and permission, also have a very complex loading models depend on logic and fallbacks.

Forcing SharePoint to use singledomain mode prevents the power of sharing loaded images and assembilies with different inter processes operations which effects performance and loading times which you know SharePoint’s warm up is already slow.

If you have more than one web application, it effects more. Even if you have only one web application, don’t forget that we have several service applications too.

And another one, this registry change may also effect your other .net applications (any kind of) because it is a machine level change.

On the other hand by workarounding this, possibly you are doing something incorrect which SharePoint worker process may not happy with it, like injecting dlls, hooking apis etc.
That may or may not cause performance issues but you may face also different abnormalities that hard to detect when the things are getting ugly.

So what we should do ?
– First you should find it out which Dll(s) causing this problem. And verify the issue resolved when you uninstall related product.
“Microsoft.EnterpriseManagement.OperationsManager.Apm.Instrumentation, Version=7.0.5000.0”
“Microsoft.EnterpriseManagement.OperationsManager.Apm.InstrumentationUtils, Version=7.0.5000.0”
– This is mostly a Third Party Monitoring tool or even Microsoft SCOM.
These kind of tools or a component belong these tools have inject their assemblies in SharePoint worker process.
And SharePoint is very special,complex blackbox what ever you say that we can not support if you do this.
For SCOM 2012/2016 we know the reason that APM component causing this.
And we have following article that you can not deploy APM for SharePoint servers. It is NOT SUPPORTED . (***)

Client-side .NET Application Performance Monitoring (APM) is not supported for SharePoint. Enabling client-side .NET Application Performance Monitoring for SharePoint can result in unpredictable application behavior and failures.

Well , i am not a SCOM expert but following action help you deploy SCOM without APM.

NOTE ! Workaround – Installing the SCOM Agent with the NOAPM=1 switch prevents the copy of the .NET APM related DLL’s as a result w3wp.exe process cannot load the problematic components which are available under C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\*.

1) Uninstall the SCOM Agent manually on the SharePoint Servers.
2) Delete the “C:\Program Files\Microsoft Monitoring Agent” folder on your SharePoint Servers.
3) Copy the SCOM Agent folder from your SCOM Management Server to the SharePoint Servers. The Agent folder can be found under C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\amd64 on your SCOM Management Servers.
4) Open an elevated cmd window and on your SharePoint Server and install the SCOM Agent manually with the NOAPM=1 switch like in the example below.
Example : msiexec /i momagent.msi NOAPM=1
5) Afterwards please kindly install your Update Rollups. The current installed Update Rollup on your SCOM Management Server is also inside the Agent folder available.
C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\amd64

May be this article also help for your SCOM agent deployments without apm.

Recently we face some other issues with SCOM , even you have not deploy APM, it is still injecting some code in SharePoint which breaking our workerprocess.
In this condition please open a ticket for MS SCOM support team.
Related DLLs was;
“C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V7.1.10184.0\PerfMon64.dll”

Another APM issue: “System Center 2016 Operations Manager APM Agent causing heap corruption in SharePoint”

I hope this helps for correct actions.