Upload with compression using lib-storage

[drochgenius]

I am using @aws-sdk/lib-storage upload method to upload large amount of data to S3 using streams, and it works great.
However some of those files are large JSON files, and I'd like to compress them using brotli before uploading them to S3.

Using the standard Node tools to do this, I did not find an easy and clean way to compress the data before sending it to S3.

Would be great if we could pass a custom transform stream in the Upload() method params to transform the stream before uploading it to S3.

Would you have any better suggestion here?

Thanks

[yenfryherrerafeliz]

Hi @drochgenius, thanks for opening this discussion. Unfortunately there is not a built-in functionality in the AWS SDK for JavaScript V3 for compressing. You may need to compress your files before they are provided as body to upload to S3. However, you need to know that if you compress your file before is stored, then it will be stored as so, which means you may need to also handle the decompression when you need to work on the original file "without being compressed".

Thanks!

[drochgenius]

Makes sense, thanks for your answer.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.