A PFE colleague from Italy, recently exposed the following situation in an internal distribution list:
From: <removed_to_protect_identities>
Sent: Wednesday, July 18, 2012 8:36 AM
To: <removed_to_protect_identities>
Subject: Report hanging when rendering on IE9
Hi there,
I am browsing a set of reports on RS2008 Sp3 with Internet Explorer 8 and IE9.
When browsing with IE8, no problem, reports complete in few seconds.
If I use IE9 the same reports hang, without completing rendering, without any error message. The SQL query completes correctly behind the scenes, but the reports never end. It is not possible to troubleshoot by querying the ExecutionLogStorage in the ReportServer db, since the reports do not end.
Is there any known issue about IE9?
Any help would be appreciated.
Thanks a lot
<removed_to_protect_identities>
MS Services Italy
I first suggested him to trace the HTTP messages exchanged between Internet Explorer and Reporting Services Web service in the scenario where the report was fully rendered and in the one that never completed, to determine whether or not the server was somehow sending back incomplete responses or some sort of malformed content that was causing that hanging like behavior on the client.
Indeed, when the hang occurred either Fiddler or the integrated IE developer tools let us realize the client had issued several GET requests to the server which were still Pending to fully receive a response.
With that, I requested the manual generation of a memory dump of the Reporting Services server process to analyze whether there was something pending to process from that interactive report request or not. A dump of the server process when the client showed that hanging state revealed the server had nothing pending to process. So, I requested the LogFile and it showed this:
rshost!rshost!12f8!07/19/2012-17:44:58:: e ERROR: HttpPipelineCallback::SendResponse(): failed writing response.
rshost!rshost!12f8!07/19/2012-17:44:58:: e ERROR: Failed with win32 error 0x10DD, pipeline=0x000000000064DD30.
0x10DD being ERROR_INVALID_OPERATION returned by HttpSrvConn::WriteAsync.
I correlated that error to the fact that customer had an antivirus installed in that machine and that some of their well-known intrusive components (like their buffer overflow protection module) were loaded into SSRS’ address space (see below), plus taking into consideration the server was in test (not in production yet) I suggested they should uninstall the antivirus, reboot the machine, and give it another try.
As most the time happens with this type of issues, customer refused to uninstall that piece of software. This time at least they accepted creating an image server in which they would also install the AV software but they would leave the AV components in an off state. Well, it’s better than nothing, but as explained to them it doesn’t guarantee that those AV components are not poking around even when their UI say they are “disabled”. AV or security software manufacturers tend to understand “Disabled” as a “I’ll continue with all my intrusive way of doing, only that if I detect something suspicious I won’t tell anyone. But I can keep being the cause of performance problems, memory leaks, or memory corruptions.“
00000000`6fa00000 00000000`6fa3f000 sophos_detoured_x64 (export symbols) sophos_detoured_x64.dll
Loaded symbol image file: sophos_detoured_x64.dll
Image path: C:\Program Files (x86)\Sophos\Sophos Anti-Virus\sophos_detoured_x64.dll
Image name: sophos_detoured_x64.dll
Timestamp: Fri Jan 14 21:09:51 2011 (4D30AD8F)
CheckSum: 00044BBC
ImageSize: 0003F000
File version: 9.5.5.9574
Product version: 9.5.5.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Sophos Plc
ProductName: Sophos Anti-Virus
InternalName: sophos_detoured.dll
OriginalFilename: sophos_detoured.dll
ProductVersion: 9.5.5
FileVersion: 9.5.5.9574
FileDescription: Sophos Buffer Overrun Protection
LegalCopyright: © 1989-2011 Sophos Plc, www.sophos.com
00000000`75500000 00000000`75511000 swi_lsp_64 (export symbols) swi_lsp_64.dll
Loaded symbol image file: swi_lsp_64.dll
Image path: C:\ProgramData\Sophos Web Intelligence\swi_lsp_64.dll
Image name: swi_lsp_64.dll
Timestamp: Tue Sep 28 16:23:30 2010 (4CA1FA62)
CheckSum: 000116BD
ImageSize: 00011000
File version: 1.0.6.0
Product version: 9.5.4.0
File flags: 2 (Mask 17) Pre-release
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Sophos Plc
ProductName: Sophos Anti-Virus
InternalName: swi_lsp
OriginalFilename: swi_lsp.dll
ProductVersion: 9.5.4.0
FileVersion: 1.0.6.0 [2038726]
FileDescription: Sophos Web Intelligence
LegalCopyright: © 2010 Sophos Group, www.sophos.com
So, on they went and, on a new Windows Server, they installed Reporting Services and all the reports the customer used, only that in that new server they kept the antivirus services disabled.
Guess what? The issue did not show up.
However, as soon as they turned the antivirus on the hanging issue was systematically reproducible. They performed such tests many times confirming that the antivirus was the cause of those weird hangs.
They also noted that that issue appeared only when browsing the reports from the server itself. When the browsing was initiated from remote clients, that caused no problem, even if the AV was installed also on them.
It looks like it was a very specific combination of ingredients that caused the issue. Something along the lines of a data stream contained in that specific HTTP request with a very peculiar pattern one of the components of the antivirus didn’t like when it came from a request from the local host.
Once demonstrated, IT Administrators in that customer decided to configure their antivirus software so that it didn’t hooked into Reporting Services activities and inexplicably tampered with some extremely specific report requests.
No matter how strong your case against an antivirus component may be, convincing a customer that they must deactivate some parts or all of its components is a tough task. By default, people tends to over trust antivirus software above anything else.
We already had KB2033238 - You may experience performance and consistency issues with SQL Server when certain modules are loaded into SQL Server address space which warned of these types of problems, but we had never seen them in the context of Reporting Services up until now.