When try to change application pool identity for a sharepoint iis site getting “keyset does not exists” error

One day you noticed that your application pools getting stopped and when try to run again it is stopping again and after a while you suspected that the problem may be caused by identity account corruption and decided that change application pool identity or reset current identity’s password but then upps your getting fallowing error.
“Keyset does not exists (Exception from HRESULT: 0x80090016)

If you get this error first you have to check your MACHINE Keys.By the way i have to say the machine key not only used by IIS or your web sites. So many process can use machine keys to encrypt or decrypt secures like strings,passwords,connection strings etc. As you noticed if some how your machine keys are changed or deleted you may have a big problem if you dont have any proper backup.

but how could that happens ? There is so many possibility here , Malwares,Group Policies,Permission issues, User mistakes , Cleanup programs, misconfigrurations.
So you may first check the fallowing
http://support.microsoft.com/kb/977754

The LOCAL SERVICE account is the service account of the IIS Web Management Service (also known as WMSvc).  This problem occurs because the LOCAL SERVICE account does not have Read access on the iisWasKey key that is located in the following folder: %ALLUSERSPROFILE%\Microsoft\Crypto\RSA\MachineKeys
The following is the file name of the iisWasKey key: 76944fb33636aeddb9590521c2e8815a_GUID
To resolve this problem, follow these steps:

  1. Locate the following folder:
    %ALLUSERSPROFILE%\Microsoft\Crypto\RSA\MachineKeys
  2. Right-click the following file, and then click Properties:
    76944fb33636aeddb9590521c2e8815a_GUID
  3. Click the Security tab, and then click Edit. If you are asked whether you want to continue the operation, click Continue. Then, the list of group names and user names that have access to this key file appears in the Permissions dialog box.
  4. Click Add. Then, the Select Users, Computers, Service Accounts, or Groups dialog box appears.
  5. Type  LOCAL SERVICE, and then click Check Names.
  6. Click OK.
  7. In the Group or user names list, click LOCAL SERVICE. Make sure that the Read check box is checked in the Permissions for LOCAL SERVICE list.
  8. Click OK.

Compare HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography MachineGuid for records in
%ALLUSERSPROFILE%\Microsoft\Crypto\RSA\MachineKeys folder .

So what if you have missing keys:
There is two option.
1) Restore missing keys from newest good backup.
2) I am sorry to say that but second option is Total recovery:
You need to reinstall  SharePoint or IIS even may need all machine. And after do that you must reset all your sharepoint managed accounts’s password.

Machine Keys are important for security and always consider to save them.If you are suspecting that your machine keys stolen you may need to reset your all passwords not for sharepoint all accounts for that machine in use and dont forget machine keys not use only by IIS.

if you think that the issue related a permission issue you may use “Process Monitor” to find out which process can not reach your data.
http://technet.microsoft.com/en-us/sysinternals/bb896645

 

Multiple application pool identity senario when using NLB with kerberos auth for Sharepoint

First of all i assume that your farm is running behind a NLB cluster and configured using kerberos authentication successfully.

Here is the scenario:

Sharepoint 2010 WFE1 :
->IP: 192.168.10.5  FQDN : wfeserver1.contoso.com  , Windows 2008 server SP2 x64 , IIS 7.0

Sharepoint 2010 WFE2:
->IP: 192.168.10.7  FQDN : wfeserver2.contoso.com , Windows 2008 server SP2 x64 , IIS 7.0

NLB:
NLB Cluster IP : 192.168.10.200   FQDN: nlb1.contoso.com

We have 2 sharepoint application running on port 80:
App1: already configured using Kerberos Auth  :
Host Header : http://istanbul.contoso.com  AppPool account : Contoso\bugra

App2 : is using NTLM (just now)
Host Header : http://ankara.contoso.com  AppPool account Contoso\postman

In order for Kerberos authentication to work we configured:
When you run IIS in a clustered environment or in a load-balanced environment, you access applications by using the cluster name instead of by using a node name. This scenario includes network load balancing. In cluster technology, a node refers to one computer that is a member of the cluster. To use Kerberos as the authentication protocol in this scenario, the application pool identity on each IIS node must be configured to use the same domain user account. To configure each IIS node to use the same domain user account, use the following command:
Setspn –A HTTP/CLUSTER_NAME domain\username
http://support.microsoft.com/kb/929650

(Note: I could able to manage kerberos authentication without defining any SPN to NLB cluster on Windows Server 2008 R2. )

Defined SPN’s:

According to  KB  : SPN for the NLB cluster name: ***
SetSPN -A HTTP/nlb1.contoso.com     Contoso\bugra
SetSPN -A HTTP/nlb1     Contoso\bugra

SPN for the cluster node:
SetSPN -A HTTP/istanbul.contoso.com    Contoso\bugra
SetSPN -A HTTP/istanbul    Contoso\bugra

What happens if I want to configure an additional web application “ankara.contoso.com” , running under a different application pool “Contoso\postman”  also running Kerberos authentication ?

What about the NLB SPNs – they have a different account. This should be a problem of a duplicate SPN for NLB .Sure it is not able to do it like this way.

Solution:
1) Create another DNS A record on NLB Cluster ip:
ex:  host  A  newnlbrecord.contoso.com 192.168.10.200

2) Create SPN for this FQDN:
SetSPN -A HTTP/newnlbrecord.contoso.com     Contoso\postman
SetSPN -A HTTP/newnlbrecord Contoso\postman

And dont forget to create for your app:
SetSPN -A HTTP/ankara.contoso.com    Contoso\postman
SetSPN -A HTTP/ankara Contoso\postman

end of article.