Enhancement: AWS S3 Support

[GaelMartins0]

Summary

This pull request introduces support for AWS S3 storage in LibreChat, providing an alternative to Firebase CDN for handling images and avatars. This enhancement aims to increase flexibility for users who prefer or are mandated to use AWS services.

Dependencies: This change requires the use of AWS SDK for JavaScript (v3), as well as configuration changes in the environment variables and librechat.yaml.

Change Type

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Testing

Prerequisites

• Ensure you have AWS credentials with permissions to manage S3 buckets.
• Install necessary AWS SDK dependencies using npm install @aws-sdk/client-s3.

Steps

• Create an S3 bucket through the AWS Console or CLI.
• Set the necessary environment variables in your .env file:

AWS_ACCESS_KEY_ID=<your_access_key>
AWS_SECRET_ACCESS_KEY=<your_secret_key>
AWS_REGION=<your_bucket_region>
AWS_BUCKET_NAME=<your_bucket_name>

• Update your librechat.yaml to configure the file strategy:

fileStrategy: "s3"

• Modify your S3 bucket policy to restrict public access while allowing necessary permissions for your application. (To be developed further with exact policy details).
• Run unit tests to ensure integration is functioning as expected.

Reproduction

• Upload an image or avatar through the LibreChat application and verify it is stored in the specified S3 bucket.
• Validate that the URL generated is a pre-signed URL with access control and that it expires appropriately.

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have commented in any complex areas of my code
• I have made pertinent documentation changes
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective or that my feature works
• Local unit tests pass with my changes
• Any changes dependent on mine have been merged and published in downstream modules.
• A pull request for updating the documentation has been submitted.

Additional Notes

The functionality mirrors the existing Firebase CDN setup, including image and avatar handling, and adds necessary configuration for S3 bucket management. There are still issues with URL generation using pre-signed URLs which have an expiration date, implying images cannot be accessed post-expiration without regeneration.

[danny-avila]

thanks!

[danny-avila]

Have not seen progress on this, let me know if you will still be working on it or need any guidance

[GerkinDev]

I would gladly give you a hand on this if needed. In the meantime, I will use it in my fork

[GaelMartins0]

Have not seen progress on this, let me know if you will still be working on it or need any guidance

Yeah, right now I'm working on another project and I don't know when I will come back to this

[mkagit]

Have not seen progress on this, let me know if you will still be working on it or need any guidance

Yeah, right now I'm working on another project and I don't know when I will come back to this

Would be awesome to have this done! Please @GaelMartins0

[MSITETOP]

I would like to propose expanding the storage support in this PR to include other providers that are compatible with the AWS S3 API. This would enhance flexibility for users who may prefer or require alternative storage solutions beyond AWS. Including support for additional providers could greatly benefit the community and broaden the usability of LibreChat across different environments.

Thank you for your work on this feature!

[marvin-roesch]

We are currently in the process of setting up LibreChat in our organization and would love to be able to use S3-compatible storage as well since our infrastructure is designed around that. To that end, I concur with @MSITETOP's point.

Actually, V2 of the AWS JavaScript SDK has been deprecated in favour of V3 and will reach its end of support by September 2025. V3 "natively" comes with support for profiles and endpoints as used e.g. with the aws CLI tool. This means that using non-AWS S3 providers would be as simple as providing the necessary configuration. The current implementation as proposed by this PR already uses s3.getSignedUrl, which is also provided by the @aws-sdk/s3-request-presigner package under the new structure and natively respects the provided endpoint configuration as well.

@GaelMartins0 What are your thoughts on switching straight to V3 of the SDK? Migrating your existing code should require relatively little effort, AFAICT.