Improve serving speed of blockchain data from AWS S3

[bokobza]

Hi,
We're using the https://github.com/frolvanya/near-lake-framework-py repo to download and index NEAR data we need.
We've found the download to be quite slow, it takes about 80-90 seconds to download 100 blocks (even without a single line of processing, just downloading).
Additionally, the download get completely stuck from time to time and nothing gets downloaded for a whole minute.

Because of these 2 issues, our indexer never seem to catch up and the gap between our last indexed block and the tip is ever-increasing.

What can we do to increase the speed?
Maybe we can use S3 transfer acceleration?
Maybe make use of compression?

Thanks

[bokobza]

Hi @khorolets, @frol, is this something you could consider? It could potentially help the speed of the download.