System.OutOfMemoryException was thrown -- how to debug?

[Jarno Leikas]

Hello,

we're hosting a number of web services in in Windows Server 2003. Lately
we've been getting daily problems with the server where the server just stops
responding to user requests (from a Winform client) and just giving a
System.OutofMemoryException exception (please see the end of the message).
Only restarting aspnet_wp.exe (ASP.NET is running in IIS 5 mode) helps.

Now because this happens only in the evenings, I'm pretty sure we're looking
at either some kind of memory fragmentation problem (see
http://blogs.msdn.com/jamesche/archive/2006/01/20/515583.aspx or an actual
memory leak).

My question is, what is the best way to start debugging this issue? My guess
is that the CLRProfiler tool could be a good starting point, as well as
looking at memory counters. Does ASP.NET produce any kind of error log that
could shed some light on which extensions (see below) are producing the
problem?

Currently we are running WSE 2.0 SP2 for DIME attachments. (That web service
is my primary suspect anyways since the problems started after a big update
where, among other things, we started using the DIME attachments).

Thanks in advance for any help! Here's the exception and a stack trace that
our client is seeing:

* There was an exception running the extensions specified in the config
file. --> Exception of type System.OutOfMemoryException was thrown.
Source : System.Web.Services
Stack trace : at
System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at
System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String
methodName, Object[] parameters)

[Steven Cheng[MSFT]]

Hello Jarno,

From your description, you have many ASP.NET webservices deployed on a
windows 2003 server (as IIS 5 process isolation mode). Recently, you begin
to get outOfMemory Exception,correct?

Based on my experience, such OOM exception is likely due to some incorrect
memory allocation that produce memory fragment. Firstly, I'd like to
confirm that whether all the ASP.NET webservice or other web applications
on the server has been configuerd as release mode (set <compilation
debug="false"> in web.config)? This is very important.

Also, since you're using the IIS5 worker process isolation model, there
will has multiple ASP.NET application instances(appdomain) hosted in a
single aspnet_wp.exe worker process, you need to determine which
application contribute mostly in the memory consumption. To do this, you
may need to isolate application one by one(run only one application each
time).

For memory leak issue in .net application, we usually need to detect
whether the leak is due to managed memory or native memory. To check this,
we can use the following two performance counters:

** process--- Private Bytes

** .NET CLR Memory --- #Bytes In All Heaps

Managed memory is included in the "Private Bytes" counter, so "Private
Bytes" will rise whether your leak is managed or native. Incidentally, this
is why one can not determine whether a leak is managed or native from Task
manager's "VMSIZE column". VMSIZE column represents Private Bytes only.

When you use the above two counters to trace memory consume of a certain
process, if the two counters diverge(meaning Private Bytes increase and
#Bytes in All Heaps" stays relatively flat), then the leak is a native one.
If two counters show the same rising trend, it is a managed memory leak.


For managed memory debugging, there has some profiler tools such as the
CLRProfiler you mentioend that can inspect manaegd objects in .NET CLR
heap. Also, if you're familar with Debugging tools for windows, you can
even use it to debug .net application with the SOS extension. Here are some
good msdn reference introducing production debugging:


#Introduction to Production Debugging for .NET Framework Applications
http://msdn.microsoft.com/library/en-us/dnbda/html/dbgch01.asp?frame=true

#Debugging Memory Problems
http://msdn.microsoft.com/library/en-us/dnbda/html/DBGch02.asp?frame=true


This tool is quite powerful for debugging production environment
application. However, due to its complexity, it is usually used by our
product support team's engineer for critial issue's troubleshoting (live
debugging or static dump analysis).

BTW, based on the complexity of the issue, it may require some thorough
debugging to get the root cause. Therefore I would suggest you consider
contact CSS product support service if you feel it an urgent issue.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead



==================================================

Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.



Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.

==================================================



This posting is provided "AS IS" with no warranties, and confers no rights.

[Jarno Leikas]

Post by Steven Cheng[MSFT]
From your description, you have many ASP.NET webservices deployed on a
windows 2003 server (as IIS 5 process isolation mode). Recently, you begin
to get outOfMemory Exception,correct?

You are correct.

Post by Steven Cheng[MSFT]
Based on my experience, such OOM exception is likely due to some incorrect
memory allocation that produce memory fragment. Firstly, I'd like to
confirm that whether all the ASP.NET webservice or other web applications
on the server has been configuerd as release mode (set <compilation
debug="false"> in web.config)? This is very important.

Does this have to do with just using web services? We're not producing any
ASP.NET web pages, a Windows Forms client is using SOAP messages to access
the server. I actually tried to find out from the web if the <compilation>
settnig has anything to do with web services, but only found it being
mentioned along with web-based applications. But if you know that this is
important also for Web Services, I'll definitely give it a go.

Post by Steven Cheng[MSFT]
application contribute mostly in the memory consumption. To do this, you
may need to isolate application one by one(run only one application each
time).

We basically need to run the SOAP interface as one application, but I think
the client could switch to using application pools as well if I requested.

Post by Steven Cheng[MSFT]
For memory leak issue in .net application, we usually need to detect
whether the leak is due to managed memory or native memory. To check this,
** process--- Private Bytes
** .NET CLR Memory --- #Bytes In All Heaps
Managed memory is included in the "Private Bytes" counter, so "Private
Bytes" will rise whether your leak is managed or native. Incidentally, this
is why one can not determine whether a leak is managed or native from Task
manager's "VMSIZE column". VMSIZE column represents Private Bytes only.

Thanks for the info, this definitely helps me figure out whether it's an
unmanaged component that's hogging memory.

Post by Steven Cheng[MSFT]
even use it to debug .net application with the SOS extension. Here are some
#Introduction to Production Debugging for .NET Framework Applications
http://msdn.microsoft.com/library/en-us/dnbda/html/dbgch01.asp?frame=true
#Debugging Memory Problems
http://msdn.microsoft.com/library/en-us/dnbda/html/DBGch02.asp?frame=true
BTW, based on the complexity of the issue, it may require some thorough
debugging to get the root cause. Therefore I would suggest you consider
contact CSS product support service if you feel it an urgent issue.

You may be right. But I think we'll try to isolate the issue a bit further.
Debugging complex issues is generally not a problem, although it's good to
know that help is also available if we can't figure this one out in due time
ourselves.

Thanks for the help so far,
Jarno

[Kim Greenlee]

Jarno,

I watched a very interesting ASP.NET Webcast awhile ago:

"MSDN Webcast: ASP.NET Under the Covers: Creating High-Availability,
Scalable Web Applications (Level 300)"
http://www.microsoft.com/events/EventDetails.aspx?CMTYSvcSource=MSCOMMedia&Params=%7eCMTYDataSvcParams%5e%7earg+Name%3d%22ID%22+Value%3d%221032298165%22%2f%5e%7earg+Name%3d%22ProviderID%22+Value%3d%22A6B43178-497C-4225-BA42-DF595171F04C%22%2f%5e%7earg+Name%3d%22lang%22+Value%3d%22en%22%2f%5e%7earg+Name%3d%22cr%22+Value%3d%22US%22%2f%5e%7esParams%5e%7e%2fsParams%5e%7e%2fCMTYDataSvcParams%5e

Here is my blog comment on a take away from that session:

"I particularly liked the information pertaining to how virtual address
space gets fragmented by small assemblies that align on 64KB boundaries; this
means that there can be a lot of unused memory between the assemblies. With
this increased risk of virtual address fragmentation, applications suffer
out-of-memory (OOM) errors but the physical memory looks fine. Very
interesting."

You may want to take a look a quick look at the webcast to see if that is
related to your problem. Sorry I don't know the actual time or slide where
that particular discussion is started.

Good luck,

Kim Greenlee

[Steven Cheng[MSFT]]

Thanks for Greg and Kim's informative inputs,


Hi Jarno,

yes, the compilation debug setting is mainly target ASP.NET web page
application, however, since webservice application also contains things
that is dynamically compiled at runtime, it is recommend that we always set
debug=false for ASP.NET application in production environment.

Sure, you can do much work through the debugger tools for windows at your
side since its a really powerful debugger on windows. BTW, I've noticed
that Greg has referenced many good blog links from Microsoft product team
engineers about sharing some knowledge and experience on production
debugging through windbg.

Anyway, please feel free to let me know if there is anything else we can
help you.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


This posting is provided "AS IS" with no warranties, and confers no rights.

[Greg Nagel]

Hi Jarno

I've had this problem before on our production webservers.

I used windbg to sovle the problem. It's a great help.

I found these links very useful:
http://blogs.msdn.com/ricom/archive/2004/12/10/279612.aspx
http://blogs.msdn.com/tess/archive/2006/02/02/523553.aspx
http://dotnetdebug.blogspot.com/2005/06/blocked-finalizer-thread.html

It was the 1st time I had used windbg, but those articles along with the
windbg/sos help made things quite easy.

Btw - turns out my problem was with user scripts inside xsl files. They were
being recompiled and reloaded into memory every time the xsl file got used.
This ended up fragmenting the memory.

Oh, I found it easier to use windbg to get a process dump of the asp.net
work process and then analyze the dump offline on my desk top.

Cheers
Greg

Post by Jarno Leikas
Hello,
we're hosting a number of web services in in Windows Server 2003. Lately
we've been getting daily problems with the server where the server just stops
responding to user requests (from a Winform client) and just giving a
System.OutofMemoryException exception (please see the end of the message).
Only restarting aspnet_wp.exe (ASP.NET is running in IIS 5 mode) helps.
Now because this happens only in the evenings, I'm pretty sure we're looking
at either some kind of memory fragmentation problem (see
http://blogs.msdn.com/jamesche/archive/2006/01/20/515583.aspx or an actual
memory leak).
My question is, what is the best way to start debugging this issue? My guess
is that the CLRProfiler tool could be a good starting point, as well as
looking at memory counters. Does ASP.NET produce any kind of error log that
could shed some light on which extensions (see below) are producing the
problem?
Currently we are running WSE 2.0 SP2 for DIME attachments. (That web service
is my primary suspect anyways since the problems started after a big update
where, among other things, we started using the DIME attachments).
Thanks in advance for any help! Here's the exception and a stack trace that
* There was an exception running the extensions specified in the config
file. --> Exception of type System.OutOfMemoryException was thrown.
Source : System.Web.Services
Stack trace : at
System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage
message, WebResponse response, Stream responseStream, Boolean asyncCall)
at
System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String
methodName, Object[] parameters)