Scalability issues of pandas_dataframe_builder and pyarrow table_builder

[leezu]

I have a pyarrow table with columns composed of 21036 chunks (21032413 rows). Storing the table or the equivalent pandas dataframe in vineyard does not succeed in reasonable amount of time (I waited a few minutes) but hangs in record_batch_builder (and the equivalent function for pandas respectively).

That's unlike the pyarrow plasma implementation, which is implemented in C++ and just takes a few seconds:
https://github.com/apache/arrow/blob/995abdc02fed412bbd947fe41a0765036dbbe820/cpp/src/arrow/python/serialize.cc#L588-L599

Do you intend to match performance with pyarrow for large tables? Or is this out of scope for the project?

[sighingnow]

There shouldn't such a big performance gap in that case. I will look into the problem.

[sighingnow]

Hi @leezu,

I have checked the problem. I think the behaviour for your case is expected.

• Without chunking, put a table/dataframe with 21032413x10 randomly generated doubles requires 1-3 seconds to vineyard. The timing should yields a similar result with putting to plasma after serialization (maybe even better).
• When arrays in the table are chunked array (in your case), the vineyard cannot do concatentation since some of the array chunks may (potential) sharing blob with other data structures. Then when putting to vineyard, it will create about hundreds of thousands of metedatas (every vineyard object (array, recordbatch, table, and blob, etc.) has a metadata), then it will be such slow. However the pyarrow's serialization will merge all chunks and there would be just one blob.

The ways vineyard stores data structures has cons as well, you could take a reference of a certain recordbatch in the table, and when deleting the table, you could still use that recordbatch but release memory of others. And two tables could share just some of recordbatches.

You could combine chunks before putting into vineyard a gain a better performance in your case.

[sighingnow]

BTW may I know the context of your task that using vineyard to store the data? We are eagerly want to listen from the community and end-users for scenarios where vineyard could works and we welcome any suggestions to make vineyard better.

Many thanks!

[leezu]

Thank you @sighingnow for investigating the issue. Combining chunks before storing in vineyard is an acceptable workaround for me. I'll try it out.

BTW may I know the context of your task that using vineyard to store the data

The context is training models on multi-gpu, with 1 process per GPU using vineyard to store the dataset in shared memory.
Previously I tried PyArrow Plasma. I'm trying Vineyard now due to 2 reasons: 1) PyArrow Plasma is apparently unmaintained and deprecated 2) pyarrow.Table.take is around 1000x slower than the corresponding pandas operation. Vineyard allows storing dataframe in shared memory, whereas plasma only supports pyarrow table.

[sighingnow]

The context is training models on multi-gpu...

Thank you @leezu. Currently the compatibility status of pandas is enough for common cases, but still needs further improvement. Feel free to post here if you meets other problems :)

[sighingnow]

Hi @leezu,

For better diagnostic the problem, may I know the schema of your table, or your dataframe?

I will check the "combine chunks" problem.

As well as "shape" of your table/dataframe.

[leezu]

Thank you @sighingnow. As you probably saw in ARROW-11463, I guess the best way to fix this in libvineyard is to use pickle.dump instead of pyarrow serialize. This avoids the Cannot write arrays larger than 2^31 - 1 in length issue.

For the shape of the table, I have a large_list<item: uint16> with 21032413 rows (where each row is a list) such that the concatenation of all lists has more than 2**32 elements

[sighingnow]

hi @leezu, the pull request for apache arrow has been merged: apache/arrow#9394.

For arrow tables, you could pa.ipc.new_stream then write_table to set the allow_64bit option. I have also commented in the arrow jira ticket.

[leezu]

Thanks @sighingnow. The problem is that vineyard internally calls pa.serialize instead of pa.ipc.new_stream. I think this should be changed at some point.

[sighingnow]

Thanks @sighingnow. The problem is that vineyard internally calls pa.serialize instead of pa.ipc.new_stream. I think this should be changed at some point.

I see. Thanks! Will be fixed ASAP.

In fact vineyard doesn't internally call pa.serialize on arrow.Table and arrow.RecordBatch. There's ad-hoc builder to transform arrow types into corresponding C++ data structure definitions.

[leezu]

Would it make sense to use pickle.dump with out-of-band data for pandas dataframe as well? Instead of the current approach which scales badly with the number of numpy arrays making up the pandas table

[sighingnow]

That does make sense and I just know the pickle 5 could be zero-copy from the arrow jira ticket. I have opened a issue to track that: #189.

In fact the bigest concern is pickle cannot zero-copy when deserializing.