Skip to content

Commit

Permalink
update Private Cluster page content (#3266)
Browse files Browse the repository at this point in the history
  • Loading branch information
AlekseyKorzik authored Feb 2, 2024
1 parent 02a881f commit e394f13
Showing 1 changed file with 40 additions and 17 deletions.
57 changes: 40 additions & 17 deletions docs/docs/setting-up/networking-instructions/private-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,28 @@ bacalhau serve --node-type requester --private-internal-ipfs --peer none
This will produce output similar to this:

```
16:34:17.154 | INF pkg/libp2p/host.go:69 > started libp2p host [host-id:QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE] [listening-addresses:["/ip4/192.168.1.224/tcp/1235","/ip4/127.0.0.1/tcp/1235","/ip4/192.168.1.224/udp/1235/quic","/ip4/127.0.0.1/udp/1235/quic","/ip6/::1/tcp/1235","/ip6/::1/udp/1235/quic"]] [p2p-addresses:["/ip4/192.168.1.224/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip4/127.0.0.1/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip4/192.168.1.224/udp/1235/quic/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip4/127.0.0.1/udp/1235/quic/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip6/::1/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE","/ip6/::1/udp/1235/quic/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE"]]
16:34:17.555 | INF cmd/bacalhau/serve.go:506 > Internal IPFS node available [NodeID:QmWg7m5G] [ipfs_swarm_addresses:["/ip4/192.168.1.224/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2","/ip4/127.0.0.1/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2"]]
```

To connect another node to this private one, run the following command in your shell:
bacalhau serve --node-type=compute --private-internal-ipfs --peer=/ip4/10.0.2.15/tcp/1235/p2p/QmPmaoCVvMsQ8xHGdM7RBtVw5PDroxCSe7iAeS75nDLHu2 --ipfs-swarm-addrs=/ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
To use this requester node from the client, run the following commands in your shell:
export BACALHAU_NODE_IPFS_SWARMADDRESSES=/ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
export BACALHAU_NODE_CLIENTAPI_HOST=0.0.0.0
export BACALHAU_NODE_CLIENTAPI_PORT=1234
export BACALHAU_NODE_LIBP2P_PEERCONNECT=/ip4/10.0.2.15/tcp/1235/p2p/QmPmaoCVvMsQ8xHGdM7RBtVw5PDroxCSe7iAeS75nDLHu2
A copy of these variables have been written to: /home/<username>/.bacalhau/bacalhau.run
```
bacalhau serve --private-internal-ipfs --peer /ip4/192.168.1.224/tcp/1235/p2p/QmWg7m5GyAhocrd8o18dtntua7dQeEHpuHxC3niRH4pnvE --ipfs-swarm-addr /ip4/192.168.1.224/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2
:::info
`bacalhau serve` output may also contain warnings about undetected GPUs
:::
To connect another node to this private one, run the following command in your shell, specifying Peer ID and ipfs swarm address of the initial private node:
```
bacalhau serve --private-internal-ipfs --peer /ip4/10.0.2.15/tcp/1235/p2p/QmPmaoCVvMsQ8xHGdM7RBtVw5PDroxCSe7iAeS75nDLHu2 --ipfs-swarm-addr /ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
To use this requester node from the client, run the following commands in your shell:
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/192.168.1.224/tcp/53291/p2p/QmdCLbe2pUoGjCzffd75U8w1LTiVpSap88rNjzXsBhWkL2
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
export BACALHAU_API_HOST=0.0.0.0
export BACALHAU_API_PORT=1234
```
Expand All @@ -40,18 +52,21 @@ export BACALHAU_API_PORT=1234
To connect another node to this private one, run the following command in your shell:

```
bacalhau serve --private-internal-ipfs --peer /ip4/<ip-address>/tcp/1235/p2p/<peer-id> --ipfs-swarm-addr /ip4/<ip-address>/tcp/<port>/p2p/<peer-id>
bacalhau serve \
--private-internal-ipfs \
--peer /ip4/<ip-address>/tcp/1235/p2p/<peer-id> \
--ipfs-swarm-addr /ip4/<ip-address>/tcp/<port>/p2p/<peer-id>
```

:::tip
The exact command will be different on each computer and is outputted by the `bacalhau serve --node-type requester ...` command
The exact command arguments will be different on each computer and will be output by the `bacalhau serve --node-type requester ...` command
:::

The command `bacalhau serve --private-internal-ipfs --peer ...` starts up a compute node and adds it to the cluster.

## Submitting Jobs

To use this cluster from the client, run the following commands in your shell:
To use this cluster from the client, run the following commands in your shell, setting the following environment variables:

```
export BACALHAU_IPFS_SWARM_ADDRESSES=/ip4/<ip-address>/tcp/<port>/p2p/<peer-id>
Expand All @@ -60,10 +75,13 @@ export BACALHAU_API_PORT=1234
```

:::tip
The exact command will be different on each computer and is outputted by the `bacalhau serve --node-type requester ...` command
The exact command arguments will be different on each computer and will be output by the `bacalhau serve --node-type requester ...` command
:::

The command `export BACALHAU_IPFS_SWARM_ADDRESSES=...` sends jobs into the cluster from the command line client.
The command `export BACALHAU_IPFS_SWARM_ADDRESSES=...` sets an environment variable, which overrides the default ipfs swarm address, allowing the command line client to interact with the cluster. You can also use `bacalhau config set` with a `node.ipfs.swarmaddresses` parameter to change the ipfs swarm address:
```bash
bacalhau config set node.ipfs.swarmaddresses /ip4/10.0.2.15/tcp/39437/p2p/QmdqAPqfeCpFFYHTK4tAV5oSUi7soZg2v6ggzJpe441zsZ
```

## Public IPFS Network

Expand All @@ -84,16 +102,21 @@ On the **first node** execute the following:

```
export LOG_LEVEL=debug
bacalhau serve --peer none --ipfs-connect $IPFS_CONNECT --node-type requester,compute
bacalhau serve \
--peer none \
--ipfs-connect $IPFS_CONNECT \
--node-type requester,compute
```
Monitor the output log for:
`11:16:03.827 | DBG pkg/transport/bprotocol/compute_handler.go:39 > ComputeHandler started on host QmWXAaSHbbP7mU4GrqDhkgUkX9EscfAHPMCHbrBSUi4A35`
```bash
11:16:03.827 | DBG pkg/transport/bprotocol/compute_handler.go:39 > ComputeHandler started on host QmWXAaSHbbP7mU4GrqDhkgUkX9EscfAHPMCHbrBSUi4A35
```


On **all other nodes** execute the following:

```
export PEER_ADDR=/ip4/<public-ip>/tcp/1235/p2p/<above>
export PEER_ADDR=/ip4/<public-ip>/tcp/1235/p2p/<host ID from the log above>
````
Replace the values in the command above with your own value
Expand All @@ -115,15 +138,15 @@ export BACALHAU_API_HOST=address-of-first-node
A private cluster is a network of Bacalhau nodes completely isolated from any public node.
That means you can safely process private jobs and data on your cloud or on-premise hosts!
Good news. Spinning up a private cluster is really a piece of cake :cake::
Good news: spinning up a private cluster is really a piece of cake :cake::
1. Install Bacalhau `curl -sL https://get.bacalhau.org/install.sh | bash` on every host
1. Run `bacalhau serve` only on one host, this will be our "bootstrap" machine
1. Copy and paste the command it outputs under the "*To connect another node to this private one, run the following command in your shell...*" line to the **other hosts**
1. Copy and paste the env vars it outputs under the "*To use this requester node from the client, run the following commands in your shell...*" line to a **client machine**
1. Copy and paste the environment variables it outputs under the "*To use this requester node from the client, run the following commands in your shell...*" line to a **client machine**
1. Run `bacalhau docker run ubuntu echo hello` on the client machine
1. That's all folks! :tada:
Optionally, set up [systemd](https://en.wikipedia.org/wiki/Systemd) units make Bacalhau daemons permanent, here's an example [systemd service file](https://github.com/bacalhau-project/bacalhau/blob/main/ops/terraform/remote_files/configs/bacalhau.service).
Please contact us on [Slack](https://bit.ly/bacalhau-project-slack/) `#bacalhau` channel for questions and feedback!
Please contact us on [Slack](https://bacalhauproject.slack.com) `#bacalhau` channel for questions and feedback!

0 comments on commit e394f13

Please sign in to comment.