Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update Private Cluster page content #3266

Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 40 additions & 17 deletions docs/docs/setting-up/networking-instructions/private-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,28 @@ bacalhau serve --node-type requester --private-internal-ipfs --peer none
This will produce output similar to this:

```
16:34:17.154 | INF pkg/libp2p/host.go:69 > started libp2p host [host-id:QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE] [listening-addresses:["/ip4/192.168.1.224/tcp/1235","/ip4/127.0.0.1/tcp/1235","/ip4/192.168.1.224/udp/1235/quic","/ip4/127.0.0.1/udp/1235/quic","/ip6/::1/tcp/1235","/ip6/::1/udp/1235/quic"]] [p2p-addresses:["/ip4/192.168.1.224/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip4/127.0.0.1/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip4/192.168.1.224/udp/1235/quic/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip4/127.0.0.1/udp/1235/quic/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip6/::1/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip6/::1/udp/1235/quic/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE"]]
16:34:17.555 | INF cmd/bacalhau/serve.go:506 > Internal IPFS node available [NodeID:QmWg7m5G] [ipfs_swarm_addresses:["/ip4/192.168.1.224/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2","/ip4/127.0.0.1/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2"]]
```

To connect another node to this private one, run the following command in your shell:

bacalhau serve --node-type=compute --private-internal-ipfs --peer=/ip4/10.0.2.15/tcp/1235/p2p/QmPmaoCVvMsQ8xHGdM7RBtVw5PDroxCSe7iAeS75nDLHu2 --ipfs-swarm-addrs=/ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ

To use this requester node from the client, run the following commands in your shell:

export BACALHAU_NODE_IPFS_SWARMADDRESSES=/ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
export BACALHAU_NODE_CLIENTAPI_HOST=0.0.0.0
export BACALHAU_NODE_CLIENTAPI_PORT=1234
export BACALHAU_NODE_LIBP2P_PEERCONNECT=/ip4/10.0.2.15/tcp/1235/p2p/QmPmaoCVvMsQ8xHGdM7RBtVw5PDroxCSe7iAeS75nDLHu2

A copy of these variables have been written to: /home/<username>/.bacalhau/bacalhau.run
```
bacalhau serve --private-internal-ipfs --peer /ip4/192.168.1.224/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE --ipfs-swarm-addr /ip4/192.168.1.224/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2
:::info
`bacalhau serve` output may also contain warnings about undetected GPUs
:::
To connect another node to this private one, run the following command in your shell, specifying Peer ID and ipfs swarm address of the initial private node:
```
bacalhau serve --private-internal-ipfs --peer /ip4/10.0.2.15/tcp/1235/p2p/QmPmaoCVvMsQ8xHGdM7RBtVw5PDroxCSe7iAeS75nDLHu2 --ipfs-swarm-addr /ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ

To use this requester node from the client, run the following commands in your shell:
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/192.168.1.224/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
export BACALHAU_API_HOST=0.0.0.0
export BACALHAU_API_PORT=1234
```
Expand All @@ -40,18 +52,21 @@ export BACALHAU_API_PORT=1234
To connect another node to this private one, run the following command in your shell:

```
bacalhau serve --private-internal-ipfs --peer /ip4/<ip-address>/tcp/1235/p2p/<peer-id> --ipfs-swarm-addr /ip4/<ip-address>/tcp/<port>/p2p/<peer-id>
bacalhau serve \
--private-internal-ipfs \
--peer /ip4/<ip-address>/tcp/1235/p2p/<peer-id> \
--ipfs-swarm-addr /ip4/<ip-address>/tcp/<port>/p2p/<peer-id>
```

:::tip
The exact command will be different on each computer and is outputted by the `bacalhau serve --node-type requester ...` command
The exact command arguments will be different on each computer and will be output by the `bacalhau serve --node-type requester ...` command
:::

The command `bacalhau serve --private-internal-ipfs --peer ...` starts up a compute node and adds it to the cluster.

## Submitting Jobs

To use this cluster from the client, run the following commands in your shell:
To use this cluster from the client, run the following commands in your shell, setting the following environment variables:

```
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/<ip-address>/tcp/<port>/p2p/<peer-id>
Expand All @@ -60,10 +75,13 @@ export BACALHAU_API_PORT=1234
```

:::tip
The exact command will be different on each computer and is outputted by the `bacalhau serve --node-type requester ...` command
The exact command arguments will be different on each computer and will be output by the `bacalhau serve --node-type requester ...` command
:::

The command `export BACALHAU_IPFS_SWARM_ADDRESSES=...` sends jobs into the cluster from the command line client.
The command `export BACALHAU_IPFS_SWARM_ADDRESSES=...` sets an environment variable, which overrides the default ipfs swarm address, allowing the command line client to interact with the cluster. You can also use `bacalhau config set` with a `node.ipfs.swarmaddresses` parameter to change the ipfs swarm address:
```bash
bacalhau config set node.ipfs.swarmaddresses /ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
```

## Public IPFS Network

Expand All @@ -84,16 +102,21 @@ On the **first node** execute the following:

```
export LOG_LEVEL=debug
bacalhau serve --peer none --ipfs-connect $IPFS_CONNECT --node-type requester,compute
bacalhau serve \
--peer none \
--ipfs-connect $IPFS_CONNECT \
--node-type requester,compute
```
Monitor the output log for:
`11:16:03.827 | DBG pkg/transport/bprotocol/compute_handler.go:39 > ComputeHandler started on host QmWXAaSHbbP7mU4GrqDhkgUkX9EscfAHPMCHbrBSUi4A35`
```bash
11:16:03.827 | DBG pkg/transport/bprotocol/compute_handler.go:39 > ComputeHandler started on host QmWXAaSHbbP7mU4GrqDhkgUkX9EscfAHPMCHbrBSUi4A35
```


On **all other nodes** execute the following:

```
export PEER_ADDR=/ip4/<public-ip>/tcp/1235/p2p/<above>
export PEER_ADDR=/ip4/<public-ip>/tcp/1235/p2p/<host ID from the log above>
````
Replace the values in the command above with your own value

Expand All @@ -115,15 +138,15 @@ export BACALHAU_API_HOST=address-of-first-node
A private cluster is a network of Bacalhau nodes completely isolated from any public node.
That means you can safely process private jobs and data on your cloud or on-premise hosts!

Good news. Spinning up a private cluster is really a piece of cake :cake::
Good news: spinning up a private cluster is really a piece of cake :cake::

1. Install Bacalhau `curl -sL https://get.bacalhau.org/install.sh | bash` on every host
1. Run `bacalhau serve` only on one host, this will be our "bootstrap" machine
1. Copy and paste the command it outputs under the "*To connect another node to this private one, run the following command in your shell...*" line to the **other hosts**
1. Copy and paste the env vars it outputs under the "*To use this requester node from the client, run the following commands in your shell...*" line to a **client machine**
1. Copy and paste the environment variables it outputs under the "*To use this requester node from the client, run the following commands in your shell...*" line to a **client machine**
1. Run `bacalhau docker run ubuntu echo hello` on the client machine
1. That's all folks! :tada:

Optionally, set up [systemd](https://en.wikipedia.org/wiki/Systemd) units make Bacalhau daemons permanent, here's an example [systemd service file](https://github.com/bacalhau-project/bacalhau/blob/main/ops/terraform/remote_files/configs/bacalhau.service).

Please contact us on [Slack](https://bit.ly/bacalhau-project-slack/) `#bacalhau` channel for questions and feedback!
Please contact us on [Slack](https://bacalhauproject.slack.com) `#bacalhau` channel for questions and feedback!
Loading