We are offering a critical on demand hotfix for an issue a couple of customers are reporting to us.

· Identifying the issue

· Preparation to install the hotfix

· Download the hotfix

· Installation of the hotfix

· Post Installation steps

· Our current recommendation to fix the issue without the hotfix

Identifying the issue:

Issue got introduced since August 2012 CU for SharePoint Server 2010.

After publishing a new page the site load times start to increase dramatically or page requests don’t finish loading at all. You will see the infamous “An expected error has occurred“-error or if the callstack is enabled and custom errors are turned off in the web.config (CallStack="true" / customErrors mode="Off") you will see a "Request timed out." error-message.

Errors you will see in the ULS log after a user has published a page and then tries to open the page / site containing the page:

System.Data.SqlClient.SqlException: Transaction (Process ID 76) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)

at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)

at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)

at System.Data.SqlClient.SqlDataReader.HasMoreRows()

at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)

at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock)

at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock)

Unexpected query execution failure in navigation query, HResult -2146232060. Query text (if available): "BEGIN TRAN DECLARE @abort int SET @abort = 0 DECLARE @EidBase int,@EidHome int SET @EidBase = 0 SET @EidHome = NULL IF @abort = 0 BEGIN EXEC @abort = proc_GetWebNavStructNodeByIds '1CA27B52-5021-4CE1-AE8A-C553F62481C5','26F979ED-9769-4F91-B29C-CC45C798D393',13730 SELECT @wssp0 = @abort END IF @abort <> 0 BEGIN ROLLBACK TRAN END ELSE BEGIN COMMIT TRAN END IF @abort = 0 BEGIN EXEC proc_UpdateDiskUsed '1CA27B52-5021-4CE1-AE8A-C553F62481C5' END "

An unexpected error occurred while manipulating the navigational structure of this Web.

Leaving Monitored Scope (PortalSiteMapNode: Populating navigation children for web: /ABCDEF). Execution Time=66179.2079302278

PortalSiteMapProvider was unable to fetch children for node

at URL: /ABCDEF, message: An unexpected error occurred while manipulating the navigational structure of this Web., stack trace:

at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)

at Microsoft.SharePoint.Library.SPRequest.UpdateNavigationNode(String bstrUrl, Int32 lNodeId, DateTime dateParented, String bstrName, String bstrNodeUrl, Object& pvarProperties, String& pbstrDateModified)

at Microsoft.SharePoint.Navigation.SPNavigationNode.Update()

at Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.UpdateSPNavigationNode(SPNavigationNode node, SPNavigationNode previous, String name, String url, String description, String target, String audience, Boolean forceCreate)

at Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.UpgradeDraftSPNavigationNode(SPNavigationNode draftNode, SPNavigationNode previous)

at Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.<>c__DisplayClass2.<UpdateNavigationNodes>b__0()

at Microsoft.Office.Server.Utilities.Security.SecurityUtilities.RunWithAllowUnsafeUpdates(SPWeb web, Action secureCode)

at Microsoft.SharePoint.Publishing.CmsSecurityUtilities.RunWithAllowUnsafeUpdates(SPWeb web, CodeToRun secureCode)

at Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.PopulateNavigationChildrenInner(NodeTypes includedTypes)

at Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.PopulateNavigationChildren(NodeTypes includedTypes)

at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(NodeTypes includedTypes, NodeTypes includedHiddenTypes, Boolean trimmingEnabled, OrderingMethod ordering, AutomaticSortingMethod method, Boolean ascending, Int32 lcid)

at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(NodeTypes includedTypes, NodeTypes includedHiddenTypes, OrderingMethod ordering, AutomaticSortingMethod method, Boolean ascending, Int32 lcid)

at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(NodeTypes includedHiddenTypes)

at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.GetChildNodes(PortalSiteMapNode node, NodeTypes includedHiddenTypes)

Preparation to install the hotfix:

Check the section “Our current recommendation to fix the issue without the hotfix” to remove unnecessary NavNodes.

Install SP1 and December 2012 CU:

http://blogs.msdn.com/b/joerg_sinemus/archive/2012/12/27/sharepoint-2010-and-december-2012-cu.aspx

Download the hotfix:

This hotfix is a COD (Critical on Demand hotfix) and not designed to be installed on every environment especially when you do not run into the issue. Our April 2013 CU will contain the hotfix as well.

More about the different hotfix packages you can find here: http://blogs.technet.com/b/office_sustained_engineering/archive/2008/07/01/office-hotfixes-to-be-delivered-on-a-defined-schedule-in-the-form-of-cumulative-updates.aspx

You can download the hotfix from this location:

http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=2767795&kbln=en-us

Installation of the hotfix:

It is the same way how you install all the other hotfixes.

After applying the preceding updates, run the SharePoint Products and Technologies Configuration Wizard or “psconfig –cmd upgrade –inplace b2b -wait” in command line. This needs to be done on all servers in the farm with SharePoint installed. You can run psconfig in parallel on all SharePoint machines.

Do not install February 2013 CU! Please wait for April 2013 CU.

December 2012 CU is the baseline for this COD and February 2013 CU cannot contain the hotfix. But the next one which comes in April 2013 will contain the code change and therefore the hotfix as well.

Version Information:

December 2012 CU: 14.0.6131.5001

This COD: 14.0.6134.5003

Post Installation Steps:

In case you want to be sure that no additional NavNodes entries are in the database please run the PowerShell script described in the section “Our current recommendation to fix the issue without the hotfix”.

Our current recommendation to fix the issue without the hotfix:

SharePoint 2010 Navigation deadlock: Multiple concurrent requests to update navigation nodes end up in deadlock

- Steps before you install the hotfix

- Steps you need one time after the hotfix has been installed

In case you run into the “NavNodes” issue which is expected only on a Farm with heavy load and lot of changes when using publishing features. One idea is to let the Administrator know which Page creates the unnecessary NavNodes and the other idea is to run a PowerShell script to remove the unnecessary NavNodes.

Our recommendation is to run the PowerShell script one time before you install the hotfix in the farm and one time after the hotfix has been installed. In case you can wait for April 2013 CU you can also use these workarounds until that time.

1. Regularly run a scheduled powershell script to automatically detect problematic pages (checks all pages on all sites/subsites of a web application) and automatically remove the duplicates 1-by-1 via Powershell using supported and publically SharePoint API.

Ø DeleteDupNodes_Quicklaunch_and_TopNavigation.txt

=> This will log to a logfile as you can see in the sript… the logging is not perfect – for every node deletion one line in the logfile will be written – logfile will be huge…. This has optimization potential !!!

ð Some customers report that the deletion of the duplicates just takes too long 30min to 120+min and cannot make use of this workaround and use the next one:

2. Regularly run a scheduled powershell script to automatically detect navigation duplicates and send a mail via SMTP to the admins reporting the problematic pages

Ø Sendmail_for_duplicates_quicklaunch_and_topnavigation.txt

ð The admins then manually move the page to the recycle bin and restore them afterwards.

ð Due to the site is already completely inaccessible you have to use SP Designer or the Manage Content and Structure wizard (sitemanager.aspx) which doesn’t load the navigation.

ð The yellow marked parts have to be correctly set by the customer before running the script:

…

function sendMail

{

param ([string]$problemPage)

$emailFrom = "youraddress@yourdomain.com"

$emailTo = "youraddress@yourdomain.com"

if($global:navType -eq "TopNavigation")

{

$subject = "ALERT: Duplicate navigation-nodes in TopNavigation found!"

}

else

{

$subject = "ALERT: Duplicate navigation-nodes in Quicklaunch navigation found!"

}

$body = "Problematic page: " + ($problemPage)

$smtpServer = "YourSMTPServer"

$smtp = new-object Net.Mail.SmtpClient($smtpServer)

$smtp.Send($emailFrom, $emailTo, $subject, $body)

}

…

File: DeleteDupNodes_Quicklaunch_and_TopNavigation.txt

# THIS CODE-SAMPLE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED

# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR

# FITNESS FOR A PARTICULAR PURPOSE.

# This sample is not supported under any Microsoft standard support program or service.

# The script is provided AS IS without warranty of any kind. Microsoft further disclaims all

# implied warranties including, without limitation, any implied warranties of merchantability

# or of fitness for a particular purpose. The entire risk arising out of the use or performance

# of the sample and documentation remains with you. In no event shall Microsoft, its authors,

# or anyone else involved in the creation, production, or delivery of the script be liable for

# any damages whatsoever (including, without limitation, damages for loss of business profits,

# business interruption, loss of business information, or other pecuniary loss) arising out of

# the use of or inability to use the sample or documentation, even if Microsoft has been advised

# of the possibility of such damages.

################################################################################################

param

(

$url = $(Read-Host -Prompt "SiteCollection Url")

)

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

$Logfile = "duplicates_log.txt"

Function LogWrite

{

Param ([string]$logstring)

Add-content $Logfile -value $logstring

}

function tryDeleteNode

{

param

(

$node,$dictionary,$nodeCollection

)

$title = $node.Title

if(!$dictionary.ContainsKey($title))

{

$dictionary.Add($node.Title,$node.Url)

}

else

{

if($dictionary[$title] -eq $node.Url)

{

if($node.Children.Count -eq 0)

{

echo " -> Deleting Duplicate Node: $title"

$nodeCollection.Delete($node)

$global:didDelete= $true

$temp = (get-date).ToString() +";"+ ($site.Url) +";"+ ($title)

echo "$temp"

LogWrite $($temp)

}

else

{

echo " -> Dupe Node $title has children, Skipping..."

}

else

{

echo " -> Duplicate title $title found, but mismatched link, Skipping..."

}

function deleteNodesRecurse

{

$nodes = @{}

foreach($node in $quickLaunch)

{

$childNodes = @{}

foreach($child in $node.Children)

{

tryDeleteNode -node $child -dictionary $childNodes -nodeCollection $node.Children

}

tryDeleteNode -node $node -dictionary $nodes -nodeCollection $quickLaunch

}

function deleteGlobalNodesRecurse

{

$nodes = @{}

foreach($node in $gnavNodes)

{

$childNodes = @{}

foreach($child in $node.Children)

{

tryDeleteNode -node $child -dictionary $childNodes -nodeCollection $node.Children

}

tryDeleteNode -node $node -dictionary $nodes -nodeCollection $gnavNodes

}

$sitecoll = Get-SPSite $url

write-host "SiteCollection: " $sitecoll.URL

foreach ($site in $sitecoll.AllWebs)

{

write-host " -> Site: " $site.URL

{

$quickLaunch = $site.Navigation.QuickLaunch

$global:didDelete = $false

deleteNodesRecurse

$pub= [Microsoft.SharePoint.Publishing.PublishingWeb]::GetPublishingWeb($site)

$gnavNodes = $pub.Navigation.GlobalNavigationNodes;

deleteGlobalNodesRecurse

}

while($global:didDelete)

$site.Dispose()

}

$sitecoll.Dispose()

File: Sendmail_for_duplicates_quicklaunch_and_topnavigation.txt