Large amount of unmanaged memory being used

[IshmaZX82]

I have a problem with a large amount of unmanaged memory being used.
The test application processes a large number of images, downloads them one by one and saves them in the selected format.
The test uses 7000 images, the average size is 128*128 pixels.
For most formats, memory consumption is approximately 300-400 megabytes.
But when choosing the Png format, memory consumption increases to 800-900 megabytes.
The profiler shows a large amount of occupied unmanaged memory.
Calling the GC.Collect method immediately after operations does not help, memory begins to be freed only after some time.

Is there any way to reduce unmanaged memory consumption?

My sample code:

            ArrayList list = new ArrayList();
            long sum2 = 0;

            DirectoryInfo di = new DirectoryInfo("wwwroot\\images\\");
            var files = di.GetFiles();
            foreach (FileInfo fi in files)
            {
                using (var img = SixLabors.ImageSharp.Image.Load(fi.FullName))
                { 
                    using (MemoryStream ms = new MemoryStream())
                    {
                        img.SaveAsPng(ms);
                        var bytes2 = ms.ToArray();
                        sum2 += bytes2.Length;
                        list.Add(bytes2);
                    }
                }
            }

Archive with used pictures:
https://www.dropbox.com/scl/fi/wwzd8vsfqmo4f86lubcmq/images.zip?rlkey=s5bqfejidpfxn9464xjwt99fu&dl=0

Profiler screenshot:

[JimBobSquarePants]

Png has to convert your pixel format into a compatible one for encoding. I would suggest using an explicit PngEncoder instance set with PngColorType.Rgb so you're not introducing an additional alpha tag.

Other than that, no not really. It's doing more work. There might be optimization opportunities but the code is there for you to improve upon.

[antonfirsov]

Png has to convert your pixel format into a compatible one for encoding

I doubt that the additional alpha encoding alone can explain the >2x difference in memory consumption. Or am I misunderstanding you?

@IshmaZX82 since you seem to be using the JB memory profiler anyways, can you do us a favor and post a memory snapshot with unmanaged allocations turned off?

Configuration.Default.MemoryAllocator = new SimpleGcMemoryAllocator();

This will turn all unmanaged allocations into managed byte[] allocations, which is needed so we can reason about object graphs in the snapshot. We don't recommend to use it in production though, because it's way less efficient. (Won't help with overall memory footprint, and will reduce throughput bc of GC stress.)

[antonfirsov]

I cannot repro this with the images and code from #2691 (comment), unmanaged memory use peaked around 20MB for me. @IshmaZX82 if the profiler screenshot is from your real app, you may be missing
Dispose / using statements.

[IshmaZX82]

@IshmaZX82 since you seem to be using the JB memory profiler anyways, can you do us a favor and post a memory snapshot with unmanaged allocations turned off?

Profiler screenshot for SimpleGcMemoryAllocator:

I cannot repro this with the images and code from #2691 (comment), unmanaged memory use peaked around 20MB for me.

I did some more tests. My problem is reproduced in the simple empty Asp.Net Core application, but not reproduced in the .Net application.

In the Asp.Net Core application I made the following trick: I call GC.Collect() after processing every 500 files, and the memory problem also disappeared.

[JimBobSquarePants]

Can you post a sample application?

[IshmaZX82]

My simple application with images:
https://www.dropbox.com/s/0pgh8rd6c0jrn68/WebApplication1.zip?dl=0

[JimBobSquarePants]

That's going to be GC settings then surely? Nothing to do with us.

https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc