About Supportability of SCOM APM Agent by SharePoint Products

For SharePoint 2013, SharePoint 2013 is in “extended support” state. Any integration of newer version of SCOM agent is out of support for SharePoint supportability perspective.
Microsoft strongly suggest upgrade to the most recent version of Sharepoint which is “SharePoint Server 2019”.
Supportability options are also available for SharePoint 2016 with builds of May CU 2018 or higher.

Please check the product life cycle
https://support.microsoft.com/en-us/lifecycle/search/935

Since November 2018, SCOM 2016 APM agent with UR6 or higher builds are supported using with Sharepoint 2016 and SharePoint 2019.
More information about Update Rollup 6 (UR6) for SCOM 2016

https://kevinholman.com/2018/10/31/ur6-for-scom-2016-step-by-step/

Be aware that SCOM build number versus Monitoring Agent version is different things.

https://buildnumbers.wordpress.com/scom/

https://blogs.technet.microsoft.com/kevinjustin/2018/09/06/mma-agent-and-scom-agent-version-numbers/

Using earlier version SCOM 2012  (including SP1 , R2 )  APM agent with any SharePoint product is “Not supported”. Because the necessary fixes are not implemented on this version.
Again SCOM product compoents may use without APM agent.

Most of the SCOM APM and SharePoint problems also related with .NET version that in use. It is impossible to test for every scenarios and builds.
Which causes limited supportability options most of the scenarios.

Our main supportability approach for Sharepoint;

Any kind of 3rd party profiler or Microsoft based profilers (which is also considered 3rd party for SharePoint perspective) we have only “Limited Support” and “Not recommended
(If the products are in supported builds otherwise it is “not supported” at all)

For Microsoft based (like APM) or 3rd party (.NET) profilers (where injects their binaries inside IIS worker processes and intercept executions in several points) is “not recommended” with SharePoint production envrionment.
Still they may be useful for pre-production to test and verify custom solution behaviours with in “Limited Support”.

SharePoint have highly complex implementation with IIS and ASP.NET which profiler may cause extra overhead and abnormalities.

SharePoint core works on native code and mostly wraps with .NET wrappers and do heavy interops which mostly .NET profilers useless for monitoring the core and very open to causes abnormalities and may crashes.

Profilers causes extra overhead already on SharePoint which is already a heavy product.

Sharepoint with combined .NET , ASP.NET and OS system has more than enough performance counters for profiling and monitoring perspective.

Profilers may mask underlying problems or hardining troubleshooting for deepdive analysis like dumps and memory traces, that increases our problem resolution times and may cause misleading directions.
Microsoft Support team mostly request to remove/uninstall the 3rd party profiler for before working related scenarios.

In Summary,

Use as much as possible for monitoring or profiling purposeses already provided performance counters by the SharePoint, .Net and OS System.
Avoid any kind of binary injecting codes/profilers inside worker processes where SharePoint runs for specially production environments.

Advertisements

Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

*** UPDATE November 2018, SCOM team has released a fix for SCOM 2016 (UR6 for SCOM 2016) and mitigated the APM issue. (If you are using SCOM 2012 unfortunately this is still not supported) Personally i do not recommend using any kind of profiler with a SharePoint production enviroment. 

More info:
https://blogs.technet.microsoft.com/kevinholman/2018/10/31/ur6-for-scom-2016-step-by-step/

** UPDATE June 2018. Recently we had another discussion with our product group about disabling LoaderOptimization. There are some rare scenarios that we have no other option that disabling LoaderOptimization as workaround. Not for APM scenarios, APM is still not supported.

More details:
https://blog.bugrapostaci.com/2019/07/01/about-supportability-of-scom-apm-agent-by-sharepoint-products/

The error when the issue happens:
Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)

Recently we have facing many issues with above exception .Effected products are SharePoint 2013 and SharePoint 2016.Mostly the cases are open as CritSit because of SharePoint down.

• We typically see these errors with non-SharePoint code running, such as SCOM’s APM agent or 3rd party monitoring software, or very unlikely .Net patches.
• Specifically in our case, we found the environment running APM monitoring Agent: APMAGT
• The most likely that the CLR was trying to optimize a reload of the instrumentation agent with an incompatible profiler loading sequence.

If there is a profiler exists, .Net Profiler api , gives a chance for a tool to examine and modify the IL before it gets to the JIT.  This allows one to add probe calls for things like code coverage, performance, or so whatever. As you imagine manuplating IL by profiler that may cause several abnormalities.

Cascade of events leading to the exception when executing the .Net Profiler callback though is in native code, instantiates managed code. All the instances in the process should be for the FullTrust Application Pool AppDomain, however since a generated call if it is not mapped to an AppDomain,(Yes it is possible, simple example IIS application pool ping feature)  it loads it on Default AppDomain which is not in FullTrust it causes a problem of loading an assembly with different permission set.

“Today only the top level assembly’s grant set is checked. If later CLR sees an incompatible security grant set for any assemblies in the binding closure, CLR will not be able to recover from the error and will fail to load the assembly in the new appdomain. Essentially the host is responsible for not violating this constraint.”
https://blogs.msdn.microsoft.com/junfeng/2004/08/05/domain-neutral-assemblies/

I have seen many misredirected articles even in technet blogs and incorrect resolutions are suggested for resolving this problem.

One of them changing legacy cas model settings.
Which is out of the box for SharePoint products the Legacy CAS model is a requirement.So we have this value in every SharePoint web sites.
trust level="Full" originUrl="" legacyCasModel="true"
(This is correct!!!)
People are resolving this problem by setting legacyCasModel=”false” which is NOT SUPPORTED .

You shouldn’t change it because of backward compatibility and dependancy reasons we need that. Disabling this property cause anomalies.

Please be aware meaning of NOT SUPPORTED ,does not mean your SharePoint not work. It may work but  we may not give support unless you rollback your changes or can not forseen any other anomalies .

Another incorrect resolution.
Create a new registry DWORD value called LoaderOptimization and give it the value 1 within the key
“HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework”
Perform IISRESET and check Web Apps behavior.

In key ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework’, create a new ‘DWORD (32-bit) Value’ named “LoaderOptimization” with a value 1 (either in decimal or hexadecimal as they are the same).
In key ‘HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework‘, create a new ‘DWORD (32-bit) Value’ named “LoaderOptimization” with a value 1 (either in decimal or hexadecimal as they are the same).

This configuration will disable the loader optimization of assemblies that provoke the aforementioned behaviour by setting assembly loading into SingleDomain mode.

And this is NOT RECOMMENDED** for SharePoint by Microsoft.

First of all, ASP.NET is using  by default LoaderOptimization attribute as MultiDomainHost. (System.Web/Hosting/ApplicationManager.cs).  It is not specific to SharePoint.
https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2008/43wc4hhs(v=vs.90)
There are some reason for why we don’t recommend this:
.
1) These settings are not tested by product group. (It is not design to work in SingleDomain mode)

2) Let’s have look what is LoaderOptimization enumeration
https://msdn.microsoft.com/en-us/library/system.loaderoptimization(v=vs.100).aspx
We have setting “LoaderOptimization” as 1 which means the “SingleDomain” mode.
“Indicates that the application will probably have a single domain, and loader must not share internal resources across application domains.”

SharePoint have many assemblies with many different locations and permission, also have a very complex loading models depend on logic and fallbacks.

Forcing SharePoint to use singledomain mode prevents the power of sharing loaded images and assembilies with different inter processes operations which effects performance and loading times which you know SharePoint’s warm up is already slow.

If you have more than one web application, it effects more. Even if you have only one web application, don’t forget that we have several service applications too.

And another one, this registry change may also effect your other .net applications (any kind of) because it is a machine level change.

On the other hand by workarounding this, possibly you are doing something incorrect which SharePoint worker process may not happy with it, like injecting dlls, hooking apis etc.
That may or may not cause performance issues but you may face also different abnormalities that hard to detect when the things are getting ugly.

So what we should do ?
– First you should find it out which Dll(s) causing this problem. And verify the issue resolved when you uninstall related product.
Like
“Microsoft.EnterpriseManagement.OperationsManager.Apm.Instrumentation, Version=7.0.5000.0”
“Microsoft.EnterpriseManagement.OperationsManager.Apm.InstrumentationUtils, Version=7.0.5000.0”
– This is mostly a Third Party Monitoring tool or even Microsoft SCOM.
These kind of tools or a component belong these tools have inject their assemblies in SharePoint worker process.
And SharePoint is very special,complex blackbox what ever you say that we can not support if you do this.
For SCOM 2012/2016 we know the reason that APM component causing this.
And we have following article that you can not deploy APM for SharePoint servers. It is NOT SUPPORTED . (***)

https://technet.microsoft.com/en-us/library/jj614617.aspx?tduid=(1dfb939b69d4a5ed09b44f51992a8b97)(256380)(2459594)(TnL5HPStwNw-v0X_tBOK3jzpbtaadMW8RA)()
Client-side .NET Application Performance Monitoring (APM) is not supported for SharePoint. Enabling client-side .NET Application Performance Monitoring for SharePoint can result in unpredictable application behavior and failures.

Well , i am not a SCOM expert but following action help you deploy SCOM without APM.

NOTE ! Workaround – Installing the SCOM Agent with the NOAPM=1 switch prevents the copy of the .NET APM related DLL’s as a result w3wp.exe process cannot load the problematic components which are available under C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\*.

1) Uninstall the SCOM Agent manually on the SharePoint Servers.
2) Delete the “C:\Program Files\Microsoft Monitoring Agent” folder on your SharePoint Servers.
3) Copy the SCOM Agent folder from your SCOM Management Server to the SharePoint Servers. The Agent folder can be found under C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\amd64 on your SCOM Management Servers.
4) Open an elevated cmd window and on your SharePoint Server and install the SCOM Agent manually with the NOAPM=1 switch like in the example below.
Example : msiexec /i momagent.msi NOAPM=1
5) Afterwards please kindly install your Update Rollups. The current installed Update Rollup on your SCOM Management Server is also inside the Agent folder available.
C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\amd64

May be this article also help for your SCOM agent deployments without apm.
https://blogs.technet.microsoft.com/kevinholman/2017/08/05/reinstalling-your-scom-agents-with-the-noapm-switch/

Recently we face some other issues with SCOM , even you have not deploy APM, it is still injecting some code in SharePoint which breaking our workerprocess.
In this condition please open a ticket for MS SCOM support team.
Related DLLs was;
“C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V7.1.10184.0\PerfMon64.dll”

Another APM issue: “System Center 2016 Operations Manager APM Agent causing heap corruption in SharePoint”
https://support.microsoft.com/en-us/help/3216459/system-center-2016-operations-manager-apm-agent-causing-heap-corruptio

I hope this helps for correct actions.