Is there a way to limit the maximum unmanaged memory allocated ?

[Socolin]

Hello,

I'm working on a worker that process images, one at the time, we have lots of image so we need to spawn the worker multiple time, most of the image we process are relatively small like ~10Mpixel but sometime we have larger image. To reduce cost we have 2 worker configuration, one worker with a lot of instance using a small memory footprint, and a few worker for the rare case where we have to process big image.

To detect large image we set up a limit on the memory allocator, which work well if I have an image that would use more than 384 MB of ram for the buffer, it fail with a nice error message

var configuration = Configuration.Default.Clone();
configuration.MemoryAllocator = MemoryAllocator.Create(new MemoryAllocatorOptions
    {
        MaximumPoolSizeMegabytes = 384,
        AllocationLimitMegabytes = 384,
    }
);
var image = Image.Load(new DecoderOptions
    {
	Configuration = configuration,
	MaxFrames = 1,
    },
    stream
);

But lately I discover that sometimes 1GB of unmanaged memory where allocated during the decompression of a JPEG file (9 000 x 12 000)

I was expecting the AllocationLimitMegabytes to limit kind of the maximum memory the full process (decoding + the image result) is using. So I could catch the exception and send the image to the worker with more memory. And avoid the process being killed by kubernetes when it use too much memory.

And maybe it's normal, I know very little about jpeg decoding, but 1GB to decode a 320MB images feels stange, is it normal ? (No need for details I just want to raise this in case this is not supposed to happen)

[Socolin]

After some testing with ImageMagick, the memory usage during decode is similar (It can get 1G to decode the image) but the memory limit ImageMagick support prevent going over the limit.

Would be nice to have an equivalent in ImageSharp

ResourceLimits.Memory = (ulong)(384 * 1024 * 1024);
ResourceLimits.MaxMemoryRequest = (ulong)(384 * 1024 * 1024);
ResourceLimits.Disk = 0;

[JimBobSquarePants]

Hi @Socolin sorry for the slow reply.

We're currently able to restrict allocation for individual memory requests however setting a maximum limit would require tracking multiple requests operationally. It's very likely that ImageMagick is making less and larger requests which hit the lower limit, but we are making more smaller requests which accumulate.

I'm not sure we could perform that accumulative tracking in a reasonable manner. If we were to do so it would have to be something that limited memory allocation across all transactions on all threads.

@antonfirsov What are your thoughts here? We already track totally allocations via MemoryDiagnostics Do you think applying such a limit is reasonable from a usage perspective or should users suse our Identify methods to weed out input that would hit our limits. I'm not convinced from a development perspective having a limit in place that could be unknowingly hit would be a great experience.

[antonfirsov]

Edit: there were several edits to this post, pls check history

If I'm reading this correctly, @Socolin's need is to throw a recognizable exception if the total memory footprint (managed or not) is over a certain limit so the exception can be caught and the request load-balanced to another node. If this feature request is common enough, we can for sure implement it, but I wouldn't do it after only one request.

For sure, there is a workaround, that might be good enough, since it's possible to calculate the maximum memory footprint of a single image via Identify. The only disadvantage is that if stream is a non-seekable stream (eg. NetworkStream) there is a need for allocating a temporary MemoryStream. That overhead can be reduced by pooling.

If the large allocations are caused by outliers, limiting single allocations might be good enough:

Untested code!

public class ImageTooLargeException : Exception { }

// Microsoft.Extensions.ObjectPool
private static ObjectPool<MemoryStream> s_objectPool = ObjectPool.Create<MemoryStream>();
    
public static async Task<Image> SafeLoadAsync(Stream stream, int maxImageMemoryMegabytes = 384, CancellationToken cancellationToken = default)
{
    MemoryStream? memoryStream = null;

    try
    {
        // This can be simplified a lot, if stream is always expected to be seekable.
        if (!stream.CanSeek)
        {
            memoryStream = s_objectPool.Get();
            await stream.CopyToAsync(memoryStream, cancellationToken);
            memoryStream.Seek(0, SeekOrigin.Begin);
        }

        ImageInfo info = Image.Identify(stream);
        int bytesPerPixel = info.PixelType.BitsPerPixel / 8;
        int imageSizeMegabytes = info.Width * info.Height * bytesPerPixel / (1024 * 1024);
        if (imageSizeMegabytes > maxImageMemoryMegabytes) throw new ImageTooLargeException();

        stream.Seek(0, SeekOrigin.Begin);
        await Image.LoadAsync(new DecoderOptions() { MaxFrames = 1 }, stream, cancellationToken)
    }
    finally
    {
        if (memoryStream is not null)
        {
            memoryStream.Seek(0, SeekOrigin.Begin);
            s_objectPool.Return(memoryStream);
        }
    }
}

If sizes like 9k x 12k are very common, you can introduce a memory-tracking aspect to your image manipulation logic, though I admit it's kinda complicated to implement at application level. IMO even if 30% of your inputs can be considered large, you may be better offloading those. If it's even higher, it might be easier to offload every image processing request.

If the inputs come from untrusted sources, you should be extremely careful, see:
https://docs.sixlabors.com/articles/imagesharp/security.html

[Socolin]

I think we only need to track unmanaged memory usage in our scenario, since the process is working on a single thread manner we are cover with the environment variable DOTNET_GCHeapHardLimit that will throw System.OutOfMemoryException if too much managed memory start being used. We can then catch this and redirect the image to a worker with more memory.

I just have a doubt about what happen if an OutOfMemoryException is throw during the decode, will everything will be disposed ? (I'll see if this happen and investigate if needed)

Having a global limit that include managed and unmanaged allocation could be useful for scenario of processing multiple image in the same process.

About the limit on the size of the image we process with Identify the problem is that PNG / JPEG decoder does not have the same memory footprint, so figure out the pixel size limit would be hard:

• I don't know what trigger more or less memory usage when decoding an image. I don't think 2 images of the same size, encoded with different method in PNG for example will use the same amount of memory
• Over time, with change to the implementation of the decoder, the memory footprint might change, so if I do extensive test to find a good limit now, nothing guarantee those value will still be acceptable in the future.

If sizes like 9k x 12k are very common, you can introduce a memory-tracking aspect to your image manipulation logic, though I admit it's kinda complicated to implement at application level. IMO even if 30% of your inputs can be considered large, you may be better offloading those. If it's even higher, it might be easier to offload every image processing request.

All image are already offloaded to workers, we just have 2 sized worker for now, we accept image that take up to 2 GB once decoded. We have very few of them so we have like 2 workers with like 4G memory, and many workers with 1G memory. If an image is "too big" for the small worker, it send it to the larger worker.

If the inputs come from untrusted sources, you should be extremely careful, see:
https://docs.sixlabors.com/articles/imagesharp/security.html

Thanks, I already took a look at this and we are already using the MemoryAllocator.Create, and we have the mitigation listed already in place. if a photo "crash" the process we have a protection to avoid to process again this photo, and the worker is only processing photos with the strict minimum set of permission.
My problem right now is that legit photo are crashing the process because they cause OOM with unmanaged memory. So those photo are marked as "bad", instead of "need more memory".

[Socolin]

So I did some experiment and I was able to successfully write an allocator that do the job (no pooling etc..).

Then I saw SimpleGcMemoryAllocator and I think maybe it can fix the issue since it would follow the hard limit DOTNET_GCHeapHardLimit=0x18000000.

Do you have any suggestion about using or not using this allocator ?

[antonfirsov]

SimpleGcMemoryAllocator might give you what you want, but it will hurt performance.

Another workaround that just came into my mind is to implement a tracking allocator that wraps MemoryAllocator.Default.

Here is a poorly tested prototype:
https://gist.github.com/antonfirsov/05ce2d02a04134ab0879a19c345ed71e

Does this look like a reasonable workaround?

[Socolin]

SimpleGcMemoryAllocator might give you what you want, but it will hurt performance.

Performance are impacted but, seems acceptable for our scenario (Decode + Resize) (https://gist.github.com/Socolin/32966cd90076f14b429bf477a0f9c083)

ImageSize: 9 000 x 12 000
| Method                  | Mean     | Error    | StdDev   |
|------------------------ |---------:|---------:|---------:|
| SimpleGcAllocator       | 596.3 ms |  4.95 ms |  4.63 ms |
| MemoryTrackingAllocator | 612.0 ms |  6.07 ms |  5.38 ms |
| DefaultAllocator        | 550.4 ms |  1.65 ms |  1.54 ms |

ImageSize: 4 000 x 3 000
| Method                  | Mean     | Error    | StdDev   |
|------------------------ |---------:|---------:|---------:|
| DefaultAllocator        | 90.98 ms | 0.187 ms | 0.175 ms |
| SimpleGcAllocator       | 94.82 ms | 1.481 ms | 1.386 ms |
| MemoryTrackingAllocator | 89.34 ms | 1.532 ms | 1.504 ms |

Another workaround that just came into my mind is to implement a tracking allocator that wraps MemoryAllocator.Default.

Here is a poorly tested prototype: https://gist.github.com/antonfirsov/05ce2d02a04134ab0879a19c345ed71e

Does this look like a reasonable workaround?

This look reasonable, I need to do more tests but either with the SimpleGcAllocator, the one you propose or another variation, I can fix my issue.

Thanks for the suggestions and the quick answers :)

[antonfirsov]

MemoryTrackingAllocator | 612.0 ms

Wow I didn't expect it to perform so poor, a built-in feature could greatly speed it up, if there will be more users requesting it we may consider the addition.

I'm glad that my answers helped!

[Socolin]

I had lots of variations between runs of the benchmarks (which is very strange with all the warmup etc ..) so I would not take those values as-is.

The values are all within an acceptable range for our scenario so I did not dig this more.