Multi tool use

How much garbage collection is too much

Our web application has several hundreds of users and is (as always) mainly consisting of legacy code.

After our move to Azure we were able to see and measure more metrics than before. We are experiencing performance issues and I see that our garbage collection is going through the roof (As measured in the diagnostics tab of the web app, under the performance counters section). In one minute we were able to have these numbers:

And this for a mere 18580 HTTP requests, so on average we have:

These numbers are still rising, even though the amount of requests stays about the same (see graph)

My question / remarks are:

Thank you very much in advance,
John

Update 1: 30/06/2018 @ 8:16 UTC+2

After updating application insights to monitor the garbage collection more closely I found a big eye-opener in performance hit. First of all, this is the average percent of time spent in GC:

It averages around 4,5 percent of the time (but there has been an inactive period during the night in this period) and the average peeks around 10% of the time. Then I thought to visualize the maximum amount of time the application was in GC mode and I almost fell of my chair:

This might be a wrong image. But it illustrates that our code has to wait for GC a lot! We really are going to have to fix this.

3 Answers
3

These numbers are suspiciously high for a typical web app. I'd say they are 10-1000x what they normally are (75110 GEN 2 in one minute. Sounds more like a micro benchmark for the GC :) ).

Is someone calling GC.Collect()? Grep the source code for that.

GC.Collect()

Still, you need to find out if they are causing your perf problem. If they are not causing it then there is no need to fix this. Look at the time spent in GC counter. You can use PerfView to easily measure the GC pauses that are taken. That get's you an idea for the pause latency that customers are facing.

Are these numbers the amount of objects that the GC cleans, or the amount of times GC had to be active?

These are GCs, not objects.

Under a load like this, how much GC's would be concidered as "normal", knowing very well that none is the perfect answer, but practically...

"None" is certainly not the right answer. There is no point in saving GCs if it does not improve performance meaningfully. You spend dev time for nothing if you go after that goal. You can certainly take a "normal" number of GCs.

There is no way to give a normal number. It's more about time spend in GC (which is the overhead you have to pay for) and the G2 pause times that customers will take waiting for the page to load.

How is it possible that even if the amounts of request stays the same, the number of GC collections are rising like this?

Something horrible sits in your code I would say :) Maybe threads continuously spinning up calling GC.Collect()?! A nightmare come true. Grep your code and report back. I will extend this answer to help your investigation.

GC.Collect()

Using PerfView or some profiler (I use JetBrains) you should be able to see where GCs are triggered in your code.

Lets learn something for Generations first:

Generation 0. This is the youngest generation and contains short-lived objects. An example of a short-lived object is a temporary variable. Garbage collection occurs most frequently in this generation.

Newly allocated objects form a new generation of objects and are implicitly generation 0 collections, unless they are large objects, in which case they go on the large object heap in a generation 2 collection.

Most objects are reclaimed for garbage collection in generation 0 and do not survive to the next generation.

Generation 1. This generation contains short-lived objects and serves as a buffer between short-lived objects and long-lived objects.

Generation 2. This generation contains long-lived objects. An example of a long-lived object is an object in a server application that contains static data that is live for the duration of the process.

Objects that are not reclaimed in a garbage collection are known as survivors, and are promoted to the next generation. Objects that survive a generation 0 garbage collection are promoted to generation 1; objects that survive a generation 1 garbage collection are promoted to generation 2; and objects that survive a generation 2 garbage collection remain in generation 2.

When the garbage collector detects that the survival rate is high in a generation, it increases the threshold of allocations for that generation, so the next collection gets a substantial size of reclaimed memory. The CLR continually balances two priorities: not letting an application's working set get too big and not letting the garbage collection take too much time.

The size of the ephemeral segment varies depending on whether a system is 32- or 64-bit, and on the type of garbage collector it is running. Default values are shown in the following table.

32-bit 64-bit


Workstation GC 16 MB 256 MB

Server GC 64 MB 4 GB

Server GC with > 4 logical CPUs 32 MB 2 GB

Server GC with > 8 logical CPUs 16 MB 1 GB

The ephemeral segment can include generation 2 objects. Generation 2 objects can use multiple segments (as many as your process requires and memory allows for).

The amount of freed memory from an ephemeral garbage collection is limited to the size of the ephemeral segment. The amount of memory that is freed is proportional to the space that was occupied by the dead objects.

Reference: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals

Sounds like you've ruled out any rogue GC.Collect() calls (unless you have a third party library misbehaving).

GC.Collect()

Given the amount of time spent in garbage collection, it is worth checking whether there is increase in the allocation rate of objects on the managed heap. You should add a performance counter in Application Insights to monitor Allocated Bytes/second. Not necessarily a problem though if you're not holding onto them.

As you say there is not an associated increase in traffic, then it is more likely to be a problem of objects surviving collections - this can increase the duration of collections. Might be some temporary objects that have remained reachable. You'll need to use a memory profiler to look more closely at this.

Worth also comparing to the memory usage over the same time periods too, to help understand the trigger for collections.

A profiler is the natural next step to get a clearer diagnosis following identification of a problem through performance counters. Particularly with such a significant issue like this, it will be relatively easy to spot where things are going wrong - otherwise I'm just chucking guesses out there. Should also be able to confirm there are no forced (GC.Collect()) collections from any libs.

GC.Collect()

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.