Any documents or demo use of the communicator for bulk data exchange?

[Zarca]

Hi,
As the document mentioned in the architecture , which is linked below,
https://v6d.io/notes/divein.html#architecture

the communicator seems to act like a data manager for the bulk exchange between different vineyard nodes

But i did not find any API or demo use of this class to exchange data
between different vineyard instance.
And it seems that i can't find this class in the source release of libvineyard

wish for your aid

[sighingnow]

Thanks for your interest on that part. As described in your question, the "communicator" in the picture is a "driver" for object migration (on need).

Part of such functionality is already available in the source repo. More of the usage of migration when integrated with vineyard will be added to the repo in the next one or two weeks. I will post in this thread at that time.

[Zarca]

Hi,@sighningnow,
It seems that now the Client do have a MigarateObject API for migrate object from remote nodes.
But still, dose the RPCclient can just retrieve the metadata (like shape,id) of remote and local object,
while can't get access to the payload Blob of the Object?

[Zarca]

HI, sighingnow:
Looking highly forward to the new feature's release :) , however, i still got 2 questions relate to it:
Q1.In my concern, Can the communicator (which is in the future release) help to get buffers(not the meta data) from remote vineyard instance?
Q2.I suppose the vineyard as a distributed and enhanced version of Apache Arrow's plasma,
So did the data stored in buffers (the Blob )are kept the same format as Apache Arrow's memory layout?

[sighingnow]

For Q1, yes. There would be a "MigrateObject" API on client.
For Q2, no. we made no assumption on the arrow's memory layout. The implementation of Blob exposes a arrow::Buffer just for convenience. See also issue #56. Users can define arbiraty data structures and vineyard just requires it can be represents as a set of blobs and a metadata (the metadata determines the interpretation of those blobs).

[Zarca]

@sighingnow ,
Hi, In my assumption. I think all data will be stored as a set of Blobs, like numpy , dataframe and
Apache Arrow array. But no matter what the data it is, it seems that the Blob is indeed a arrow::Buffer.
I intended to assume the Blob is just byte array in C++, but if the Blob is actually a instance of arrow::Buffer, can we say that :
1.all data in the shared memory of vineyard are finally stored as arrow:Buffer ?

2.And did the arrow:Buffer have anything to do with the Apache Arrow?

[Zarca]

Yes, I appreciate your help and I'm really interested in defining my own data format stored into vineyard,

As for the Q2 , So you mean that the Blob can contain not only arrow's data but also any other data defined by my self?

And as I walk through the source code of the latest release(0.1.6), I noticed that the document demo shows in the link below:
https://v6d.io/notes/divein.html#vineyard-object
indicate that the str type is stored as a vineyard:Blob, however, in the source :
https://github.com/alibaba/libvineyard/blob/02f7a47286d7d77448f934707fbe50154b360693/python/vineyard/data/base.py#L143
seem to show the str type is stored as a vineyard:Scalar(which is also the result on my own computer )?
Sorry , but i doubt whether it is a temporary bug or maybe i just misunderstood it.

Forget it, Maybe the latest document is just not consistent with the latest vineyard :)...

[sighingnow]

That is not a bug, that is just an optimization in implementation. You know, strings may be usually small, and could be fit into the metadata.

You could refer the implementation of Tensor: builder/resolver in python and corresponding implementation in c++ for how to sharing abstracted data structures between different languages and runtimes. See also the implementation of DataFrame as well.

The documentation at https://v6d.io is kept to be consistent with our codebase of the main branch.

[sighingnow]

I have moved this thread to the discussion category :)