Deprecate IPFS as default publisher

[wdbaruni]

The Problem

Today, when users run bacalhau serve, an embedded and private ipfs node is also created within bacalhau process. The original intention to do this was for people to easily test bacalhau and to be self contained. Though it was not intended for production use or actual use outside of just trying out bacalhau. The recommended way was for people to run their own ipfs node outside of bacalhau, and connect to it using bacalhau serve --ipfs-connect <addr>

In addition to that, when users submit a job using bacalhau docker run without defining --publisher flag, the CLI will use ipfs as the default publisher and the job will fail if the compute node does not support ipfs.

There are many problems with our current approach, including:

1. Bacalhau compute nodes are acting as both compute and storage nodes. Even if we replace ipfs with something else, we should avoid coupling those together, and just let bacalhau integrate with different storage services as a client and not embedded. This enables bacalhau to work well with existing data as well, simplifies the operations for users, and gives operators the choice of deploying storage on the same machines as the compute nodes, or just next to them on the same rack or region.
2. ipfs is meant for public networks and we had to take extra steps to make it work as a private network. Still, I am confident it is not yet secure enough with many potential leaks, such as when an authorized node join the private network but the also public peers, it will make all the private data accessible to the public network (I think)
3. I am not sure what value ipfs is providing for our users and our use case. ipfs solves content discovery and routing, but we already know where the data is and where to get it from as we keep track of the compute node that handled the job and generated the results. If someone is already using ipfs or have other use cases to use ipfs, then great and we will continue to support an integration with ipfs as a client using --ipfs-connect, but no value to introduce ipfs to fresh new users as the default and recommended storage for bacalhau

The Proposal

1. Option 1: My first recommendation is to not provide any default publishers. If a user runs a job that prints result to stdout, then they can use bacalhau job logs to get the output (I understand logs need some improvements). If they want to publish results to a remote destination, then they can select the uri where to publish the results (e.g. s3://..., ipfs://), and the job will be routed to the compatible compute nodes. This means that bacalhau serve without any flags or configurations will support jobs that don't publish results to remote destinations, which is more than enough for testing out bacalhau. In the future, we discussed an improvement where the requester node can populate job defaults such that the requester can set a default publisher to be some s3 bucket, and now users don't need to specify the remote destination anymore in their job submission
2. Option 2: Implement a new local publisher that keeps results locally on the compute nodes, and make them accessible through a simple HTTP based API to send local results from compute nodes directly back to the client. This should enable bacalhau serve to work out of the box, and for bacalhau get to be able to retrieve results not published to stdout. The critical point that I am hoping to make clear is this should only be for testing purposes and to simplify trying out bacalhau, and shouldn't be used for actual workloads. Bacalhau is not a storage service, we don't do replication, storage monitoring, or moving data around when a node is shutting down. To avoid doing the same mistake with embedded ipfs becoming more than for testing purposes, we should:
   1. Print a warning when a user calls bacalhau get when data is using local storage
   2. The requester will NOT do any tunneling or act as a gateway to retrieve data from compute nodes and forward them to the client. Meaning the client must be on the same network as the compute node to be able to fetch the results, which should be acceptable for someone trying out bacalhau. This limitation not only avoids additional complexities to the requester node, but explicitly making it clear to not use this publisher for production workloads

I've discussed this with @aronchick before, and he wasn't inclined with the first option. I am putting both options here for completeness and to make sure the tradeoffs are clear.

Priority

• Priority: 1 (High)
• Quarter: 2024Q1
• Justification:
  • ipfs is causing a lot of pain and will be great to deprecate as soon as we can
  • we are launching on marketplace soon, and we need to figure out what else we need to deploy on our nodes and what services users need to be aware of.
  • we are not storing results in a very secure way

Tasks

Preview Give feedback

1. https://github.com/bacalhau-project/expanso-planning/issues/464
2. https://github.com/bacalhau-project/expanso-planning/issues/465
3. https://github.com/bacalhau-project/expanso-planning/issues/466
4. https://github.com/bacalhau-project/expanso-planning/issues/467
5. https://github.com/bacalhau-project/expanso-planning/issues/491
6. https://github.com/bacalhau-project/expanso-planning/issues/493
7. https://github.com/bacalhau-project/expanso-planning/issues/492
8. https://github.com/bacalhau-project/expanso-planning/issues/500
9. https://github.com/bacalhau-project/expanso-planning/issues/514
10. https://github.com/bacalhau-project/expanso-planning/issues/519
11. Remove embedded IPFS node #3816
    comp/publisher/ipfs type/epic +2
    
12. Document need to supply your own IPFS node docs#28
    +0
    

[rossjones]

@wdbaruni for a user who may have signed up via their organisation's expanso-managed account, the publisher would most likely be defined organisation-wide. Previously when I've been through security audits, a lot of the concerns are that data might be exfiltrated from the cluster and in those situations it would potentially be useful for us to disallow users from specifying their own publisher, and rely solely on the organisation's configured publisher.

[rossjones]

IPFS is no longer the default publisher, the default being Noop but settable. Any chances we want to the local publisher we can put into other tickets.

[rossjones]

IPFS node still embedded, re-opening to make sure we remove it

[rossjones]

Have updated #3816 with recommendation for making this change over two releases.

[frrist]

I am going to close this issues as IPFS as a default publisher has been deprecated in favor of the local publisher. The remaining items of work can be completed outside of this issue that has grown into an epic.