Skip to content

Commit

Permalink
Fix the out of date part in the doc. (#1938)
Browse files Browse the repository at this point in the history
Fixes part of #1936

Signed-off-by: Ye Cao <[email protected]>
  • Loading branch information
dashanji authored Jul 18, 2024
1 parent 9055b1a commit 0f8e617
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 17 deletions.
22 changes: 7 additions & 15 deletions docs/notes/architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,10 @@ three primary components:
3. The **IPC/RPC servers** manage the IPC/RPC connections from Vineyard
clients for data sharing.

Specifically, the client can obtain the metadata of the data stored in Vineyard through
both IPC and RPC connections. However, to access the data partition, the client must connect
to the Vineyard instance via the UNIX domain socket, as the data
sharing occurs through the system call of memory mapping, which requires the client to be on
the same machine as the Vineyard instance.
Specifically, the client can retrieve both metadata and data partitions stored in
Vineyard via IPC and RPC connections. However, when both connection types are available,
IPC connections are prioritized due to the greater efficiency of memory mapping for data
sharing.

.. _client-side:

Expand All @@ -64,8 +63,9 @@ routine reuse (e.g., I/O drivers). More specifically,
2. The **RPC client** communicates with *remote* Vineyard instances by connecting
to the TCP port that the Vineyard daemon is bound to.

Unlike the IPC client, the RPC doesn't allow memory-sharing between processes
but is useful for retrieving the metadata of objects in the Vineyard cluster.
Unlike the IPC client, the RPC client doesn't allow memory sharing between processes
and is less efficient in that regard. However, it is still useful for retrieving both
metadata and payloads from the Vineyard cluster via TCP or RDMA.

3. The **builders and resolvers** for out-of-the-box high-level data abstractions
offer a convenient way for applications to consume objects in Vineyard and
Expand Down Expand Up @@ -180,11 +180,3 @@ Once a Vineyard object is created and sealed in the Vineyard instance, it
becomes immutable and can NOT be modified anymore. Thus, Vineyard is not
suitable for use as a data cache to store mutable data that changes
rapidly along the processing pipeline.

*NO* instant remote data accessing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The partitions of distributed data are stored distributedly in corresponding
Vineyard instances of the cluster. Only the client on the same machine can access
the data partition. In order to access a remote partition, data migration APIs of
Vineyard can be invoked to trigger the migration process, but not for instant accessing.
5 changes: 4 additions & 1 deletion docs/notes/developers/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ concerns, please feel free to `open an issue`_ or `post it to discussions`_.
the object is local or remote. This capability enables users and internal operators to
examine essential information (e.g., chunk axis, size) about an object, assisting in
decision-making processes related to object management (e.g., determining the need for
repartitioning, planning the next workload).
repartitioning, planning the next workload). With this capability, the `vineyard client`_
connects to the Vineyard daemon that stores the object's payloads, retrieves these payloads,
and assembles the objects locally.

8. *How does migration work in vineyard? Is it automatically triggered?*

Expand Down Expand Up @@ -119,3 +121,4 @@ concerns, please feel free to `open an issue`_ or `post it to discussions`_.
.. _post it to discussions: https://github.com/v6d-io/v6d/discussions/new
.. _guide: ../../tutorials/kubernetes/using-vineyard-operator.rst
.. _command line tool: ../../notes/cloud-native/vineyardctl.md
.. _vineyard client: ../../notes/references/python-api.html#vineyard.Client
7 changes: 6 additions & 1 deletion docs/notes/key-concepts/data-accessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,14 +293,19 @@ In the illustration at the beginning of this section, we demonstrate that vineya
distributed objects partitioned across multiple hosts. Accessing these distributed objects
in vineyard can be achieved through two distinct approaches:
- Inspecting metadata using the :code:`RPCClient`:
- Inspecting metadata of global.
The metadata of global objects can be examined using the :class:`vineyard.RPCClient`. This allows
computing engines to understand the distribution of partitions of global tensors using the
RPCClient, and subsequently schedule jobs over those chunks based on the distribution information.
Mars employs this method to consume distributed tensors and dataframes in vineyard.
Additionally, by leveraging the metadata of global objects and the server metadata accessible
via :meth:`vineyard.Client.meta`, multiple RPC clients can connect to retrieve the corresponding
blobs from different nodes. These blobs are then assembled locally into a single object. This approach
is also the default method for :class:`vineyard.client`.
- Accessing local partitions of global objects using the :code:`IPCClient`:
Another prevalent pattern for accessing shared global objects involves launching a worker on each
Expand Down
4 changes: 4 additions & 0 deletions docs/notes/references/python-api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ Vineyard client

.. autofunction:: connect

.. autoclass:: Client
:inherited-members:
:members:

.. autoclass:: IPCClient
:inherited-members:
:members:
Expand Down

0 comments on commit 0f8e617

Please sign in to comment.