4.7MB PNG crashes (both with & without transformations)

[bs-thomas]

I have this weird issue. It could be related to Sharp, but I was wondering if there could be better and more performant configuration set for this library. I'm scratching my head off as I did a few comparisons, and still not understanding why it has to crash.

Here you can see the two scenarios:

(1) https://codebase-images-local.beam.style/public-image/test-big.jpg

test-big.jpg
33.40MB
JPEG IMAGE

WORKS! Automatically resizes to w=2560 (my settings.yml configured at MAX_IMAGE_WIDTH: 2560)

(2) https://codebase-images-local.beam.style/public-image/test-big-png.png

test-big-png.png
4.74MB
PNG IMAGE

CRASHES! 502 Internal server error. No idea why. Even when I set timeout: 10, memorySize: 3000 <-- Pretty high settings already.

Here are my serverless.com logs for this invocation (scenario 2):

02:16:59 pm
START RequestId: aabb3d51-4a7f-42f7-b24d-60e7fdec445f Version: $LATEST
02:17:03 pm
[ERROR] [1586931423231] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413.
02:17:03 pm
END RequestId: aabb3d51-4a7f-42f7-b24d-60e7fdec445f
02:17:03 pm
REPORT RequestId: aabb3d51-4a7f-42f7-b24d-60e7fdec445f	Duration: 3411.75 ms	Billed Duration: 3500 ms	Memory Size: 3000 MB	Max Memory Used: 357 MB	

In conclusion, 33.40MB JPG can load without issues (even with transformations).
But, a 4.74 PNG crashes.

Does anyone have any ideas on this?

[Mosnar]

@bs-thomas The issue is probably because pngs are passed through pngquant for optimization. Prior to 2.0.2, this would happen regardless of whether you wanted it to or not. We added a parameter to tweak the level of compression pngquant will apply in the 2.0.2 release. Can you ensure you're running that version?

[bs-thomas]

Thanks for your assistance @Mosnar.

After some investigation, I noticed that it was because I have tweaked the max image size from:
MAX_IMAGE_WIDTH: 2000
MAX_IMAGE_HEIGHT: 1000

To:
MAX_IMAGE_WIDTH: 2560
MAX_IMAGE_HEIGHT: 1440

The interesting thing here is, it's not because of PNGQuant at all. I think your tweak was good, but let's take a look at my log below (which I didn't notice earlier).

No older events found at the moment. Retry.
(c14d527d-6a06-499a-bef9-51166497bde6) Extended Request Id: LBwQsG2JIAMF08Q=
(c14d527d-6a06-499a-bef9-51166497bde6) Verifying Usage Plan for request: c14d527d-6a06-499a-bef9-51166497bde6. API Key:  API Stage: i9u1sdp7e1/local
(c14d527d-6a06-499a-bef9-51166497bde6) API Key  authorized because method 'GET /{any+}' does not require API Key. Request will not contribute to throttle or quota limits
(c14d527d-6a06-499a-bef9-51166497bde6) Usage Plan check succeeded for API Key  and API Stage i9u1sdp7e1/local
(c14d527d-6a06-499a-bef9-51166497bde6) Starting execution for request: c14d527d-6a06-499a-bef9-51166497bde6
(c14d527d-6a06-499a-bef9-51166497bde6) HTTP Method: GET, Resource Path: /public-image/test-big-png.png
(c14d527d-6a06-499a-bef9-51166497bde6) Method request path: {any=public-image/test-big-png.png}
(c14d527d-6a06-499a-bef9-51166497bde6) Method request query string: 
{}
(c14d527d-6a06-499a-bef9-51166497bde6) Method request headers: {sec-fetch-mode=navigate, sec-fetch-site=none, Accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9, CloudFront-Viewer-Country=SG, CloudFront-Forwarded-Proto=https, CloudFront-Is-Tablet-Viewer=false, CloudFront-Is-Mobile-Viewer=false, User-Agent=Amazon CloudFront, X-Forwarded-Proto=https, CloudFront-Is-SmartTV-Viewer=false, Host=i9u1sdp7e1.execute-api.us-east-1.amazonaws.com, Accept-Encoding=gzip, sec-fetch-user=?1, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-5e96fa6a-c7b9e3d8245c71c742098980, Via=2.0 a55558c6b6748e578253e36b174f0b2f.cloudfront.net (CloudFront), 1.1 e8cd61c9b2a785e4fc8167b0177016b8.cloudfront.net (CloudFront), upgrade-insecure-requests=1, X-Amz-Cf-Id=8nnXiSgklwOGYmyjteZTjY-bb2pVVhPj21UmO7nfqhuJtROlDqHEtg==, X-Env-Cloudfront-Default-Ttl=300, X-Forwarded-For=223.16.111.79, 13.228.69.123, 204.246.166.143, CloudFront-Is-Desktop-Viewer=true, sec-fetch-dest=document}
(c14d527d-6a06-499a-bef9-51166497bde6) Method request body before transformations: [Binary Data]
(c14d527d-6a06-499a-bef9-51166497bde6) Endpoint request URI: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-1:970164395835:function:w01-CodebaseBeamStyle-ImageHandler-local-index/invocations
(c14d527d-6a06-499a-bef9-51166497bde6) Endpoint request headers: {x-amzn-lambda-integration-tag=c14d527d-6a06-499a-bef9-51166497bde6, Authorization=************************************************************************************************************************************************************************************************************************************************************************************************************************1d6521, X-Amz-Date=20200415T121330Z, x-amzn-apigateway-api-id=i9u1sdp7e1, X-Amz-Source-Arn=arn:aws:execute-api:us-east-1:970164395835:i9u1sdp7e1/local/GET/{any+}, Accept=application/json, User-Agent=AmazonAPIGateway_i9u1sdp7e1, X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHQaCXVzLWVhc3QtMSJGMEQCIHB02hjeSl7un9OfXZdAaWkRDtTs+KwX0DjRs+XUWpGPAiA26U5qcSAh1izI4O5qwarUlAqR3lGFuwTkb+PZyu1QfSq9AwiN//////////8BEAAaDDM5MjIyMDU3NjY1MCIMLZ18XwkPZqk/893aKpEDO7TCBMKveJHSWFOLu1MwJu4v+McVXZqR+Mz0qt4NDCHDwtMY4yJ7NPDBMPOtL5w1p885pyS/eVGndCUs83hj9XyZAr2ixEijmgGrUZUob7d4jrhTGVfiQuLLExXdoOWA7VeFyagywc3EwBD0KQDW0pq2dyIbNdBVb6 [TRUNCATED]
(c14d527d-6a06-499a-bef9-51166497bde6) Endpoint request body after transformations: {"resource":"/{any+}","path":"/public-image/test-big-png.png","httpMethod":"GET","headers":{"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","Accept-Encoding":"gzip","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-Country":"SG","Host":"i9u1sdp7e1.execute-api.us-east-1.amazonaws.com","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","User-Agent":"Amazon CloudFront","Via":"2.0 a55558c6b6748e578253e36b174f0b2f.cloudfront.net (CloudFront), 1.1 e8cd61c9b2a785e4fc8167b0177016b8.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"8nnXiSgklwOGYmyjteZTjY-bb2pVVhPj21UmO7nfqhuJtROlDqHEtg==","X-Amzn-Trace-Id":"Root=1-5e96fa6a-c7b9e3d8245c71c742098980","X-Env- [TRUNCATED]
(c14d527d-6a06-499a-bef9-51166497bde6) Sending request to https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-1:970164395835:function:w01-CodebaseBeamStyle-ImageHandler-local-index/invocations
(c14d527d-6a06-499a-bef9-51166497bde6) Received response. Status: 200, Integration latency: 3435 ms
(c14d527d-6a06-499a-bef9-51166497bde6) Endpoint response headers: {Date=Wed, 15 Apr 2020 12:13:34 GMT, Content-Type=application/json, Content-Length=155, Connection=keep-alive, x-amzn-RequestId=e293baeb-9019-43f2-8995-25d5d1da3e33, X-Amz-Function-Error=Unhandled, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5e96fa6a-c7b9e3d8245c71c742098980;sampled=0}
(c14d527d-6a06-499a-bef9-51166497bde6) Endpoint response body before transformations: [Binary Data]
(c14d527d-6a06-499a-bef9-51166497bde6) Lambda execution failed with status 200 due to customer function error: Response payload size (6506812 bytes) exceeded maximum allowed payload size (6291556 bytes).. Lambda request id: e293baeb-9019-43f2-8995-25d5d1da3e33
(c14d527d-6a06-499a-bef9-51166497bde6) Method completed with status: 502
(c14d527d-6a06-499a-bef9-51166497bde6) AWS Integration Endpoint RequestId : e293baeb-9019-43f2-8995-25d5d1da3e33

It seems to me that it was failing because:

Response payload size (6506812 bytes) exceeded maximum allowed payload size (6291556 bytes).

This brings up a very interesting question.

@Mosnar Regardless of PNGQuant implementations, the current serverless-sharp implementation would crash if the file size is big, because it would hit the response payload cap at Lambda 6MB, am I correct? If so, any thoughts about what could be done? (I actually have clients who does art auctions, and they could upload images as big as 30MBs!)

[Mosnar]

Interesting! Thanks for digging into it! I never realized such a limitation existed. It sounds like we'll need to cache optimized images that are too large to serve from lambda in S3 and 301 redirect to the result in either a public S3 directory, signed URL, or CDN in front of that S3 bucket.

What do you think?

[bs-thomas]

@Mosnar Yes indeed I'm also thinking something along those lines. The one thing I haven't had a chance to verify is if I do an <img src> on an image, and it gets redirected (301 or 308), will it actually load up fine. <-- verified it seems to work (on Chrome at least)

I found this article which does something very similar. While there is sample code, there is no sample github repo.
https://medium.com/dev-cubica/resizing-images-on-the-fly-with-aws-4bb75371e795

Assuming that we have this image file in the bucket:
bucketname/some-directory/test.jpg

I think the flow can go something like this:

1. User makes request to CF (which is connected to an S3 origin)
2. Lambda edge kicks in, checks if there is query string and valid signature. Passthrough to s3 if no query string.
3. If there is query string, hashes the MD5 string, tries to open the following URI from s3: /bucketname/some-directory/.thumbnails/test.jpg.(the-md5-hash)
4. If the thumbnail exists, perfect, it should be delivered and Cloudfront will cache the response.
5. If the thumbnail does not exist, another lambda kicks in, getObject from s3, and putObject the thumbnail back into the .thumbnails directory.
6. After 5, 308 redirect to the exactly same CDN URL, so the user hits the cloudfront again, and this time, the thumbnail should exist, and cloudfront should cache the response.

Does the above make logical sense to you? I'm just imaging things, not sure how well it works.