Handling large amount of data via S3

[DanielvdC]

Hi,

I have got various .hdf5 files stored on my bucket at location:

s3://tvi-improvements/analysis/

Names of the files are in dates, like this:
2020_03_01.hdf5, 2020_03_02.hdf5, 2020_03_03.hdf5, etc.

Ideally I would loop through all these files, concatenate their data, perform analyses and visualise some columns.
I've read through the docs and it seems I should use something like this:
df_from_s3 = vaex.open('s3://tvi-improvements/analysis/*.hdf5')

However this gives the following error:
FileNotFoundError: [Errno 2] No such file or directory: 's3://tvi-improvements/analysis/2020_03_01.hdf5'

Any ideas how I can fix this issue?
cheers

[DanielvdC]

It turns out that the problem was the .hdf5 extension. Vaex works perfectly fine with snappy.parquet files in this particular set up, which is something I can use as well.

[JovanVeljanoski]

I wonder if this PR will address this issue that you are having..
#1739

[maartenbreddels]

Still a weird issue, should we convert this to an issue and get to the bottom of it?

[JovanVeljanoski]

Yeah, let's do that.