Converting vaex dataframe to csv properly.

[Hasham04]

I currently have this use case where I have 72 columns and about 20 million rows I am reading the file as a parquet from s3 doing some transformations and I need to have the dataframe in a csv format to upload to S3 . I have major concerns about RAM. I want to avoid using using pandas. What do u think is the best way to do that.thankyou.

[JovanVeljanoski]

Hi. What is your problem exactly? Vaex should help you do what you need without much memory concerns, as it can lazy load parquet files.

[Hasham04]

Thankyou for your response. I was just wondering how I should convert the dataframe to csv without having any ram issues. Is export_csv the proper way to do that because I read it was a wrapper around the to_csv from pandas. Which is why I was concerned.i am sorry if this is a stupid question I am a beginner.

[JovanVeljanoski]

While it is true that export_csv is a wrapper around to_csv from pandas, it does support export in chunks. So there are no worries about RAM there. You can specify the chunk size , which in it self will fit in memory, and will append to the same csv file.
This is all handled by vaex automatically, you just need to set the chunk size parameter. This is all documented by the way.

There may be some performance considerations, given the actual exporting is indeed done by pandas. This is discussed here:
#1687

and so far the status is unchanged.

[Hasham04]

Thankyou for your help.