AWS SDK for pandas Lambda Layer size

[k-chainalysis]

Ever seen this message?

Layers consume more than the available size of 262144000 bytes (Service: Lambda, Status Code: 400, Request ID: ...

This is what you see if you try to use the (fka) AWS Wrangler Lambda Layer plus an additional heavy layer, like OpenTelemetry or DataDog. Often a Lambda function needs only a small piece of all that delicious wrangler functionality, so I wonder if it would be possible to modularize the AWS SDK for pandas and split the Lambda Layer into smaller layers, à la carte?

It seems as this open source project grows, the size of the Lambda Layer will increase. Potentially I could try to upgrade to a new Layer version and find that I've bumped up against the layer size limit.

[LeonLuttenberger]

Hey,

We have encountered this error, yes. An issue for us here is how to reduce the layer size while keeping all our crucial dependencies. Out of the 154 MiB that our Lambda layer takes up (Python 3.9, x64, version 2.17), this is breakdown of our largest dependencies:

1. 28 MiB for pandas
2. 50 MiB for numpy (dependency for Pandas)
3. 47 MiB for pyarrow (dependency for Pandas if we want to load Parquet files, used also for schema verification in a few places)
4. 10 MiB for lxml (needed for Redshift connector)

As you can see, the main problem is that even if modularized, Pandas with it's dependencies is still responsible for the bulk of the layer size.

I will speak to the team, however, to see if they have any other ideas for what we can do.

Best regards,
Leon

[k-chainalysis]

Makes sense, thanks for the consideration. Maybe the AWS Lambda service team can carry the burden... Although I noticed they removed requests from the runtime, which seems like a backwards step...

[JHLi22]

where to find the awswrangler lambda layer arn? I saw it was deprecated. Is there any substitute

[LeonLuttenberger]

@JHLi22 We recently rebranded the library as AWS SDK for pandas. As a result, we also started publishing new versions of the Lambda layer under the new name. You can find more information in our install guide. You can also find the ARNs of the managed layers here.

[cponfick]

@LeonLuttenberger One big issue for me is the reliance on the provided lambda layers. In one of my projects I am required to install a package that has numpy as a dependency. Hence, numpy is included twice. Once in the awswrangler layer and once in my own layer. Is it possible to create an awswrangler package that can be added via pipenv, poetry, etc. that has the same package size as the lambda?

[LeonLuttenberger]

Hey @cponfick-bhs,

Are you asking if you can build a custom layer with awswrangler as one of the dependencies? AWSWrangler is primarily a Python package, so you're welcome to simply add it as a dependency in your poetry configuration. This would allow you to add it to your custom layer.

We also provide instructions on how install AWS SDK for pandas as a custom layer. You can use this as a starting point, and then add whichever extra dependencies within that.

Let me know if this answers your question.

Best regards,
Leon

[cponfick]

If I simply add it as a dependency the resulting layer will have ~300 MB, because of pyarrow.
If I add it as a custom layer it will have only ~170 MB, but if I add a package with numpy as a dependency and create my own layer I am already at a combined 230 MB.

I am not sure how I would be able to add my packages to the provided AWS SDK for pandas layer.
Is the following possible?

1. Download the AWS SDK for pandas layer via boto3
2. pip install into the layer
3. Re-upload as new layer

I think the underlying problem is that the available .zip storage in AWS lambda is simply too small. Even if I find some type of workaround I would still break the best practice of adding boto3 to my layer, since only boto3 and AWS SDK for pandas would already be 230 MB.

[mahi80]

Anyone tried to use zappa ?