Bacalhau project report 20221014
Amazing progress by the team this week, feels like we are really rolling!
Kai has made significant progess discussing FIL+ integration with the community and now has a concrete design in his head, which will soon be written up! This has been a non-trivial undertaking to figure out a path forwards, so great work on that Kai!
Kai has also been a prolific bug-fixer this week, nailing:
- fixed wait flags bug
- fixed publish on error bug
- fixing downloading urls in parallel with backoff/retry
Will has landed his first major contribution to the core codebase with a change that includes Lotus in the Bacalhau devstack. This will be extremely valuable as we deepen our Filecoin integration going forwards.
WebAssembly System Interface (WASI) specifies a standard set of syscalls that act as a target for other programs. It is only in preview (and in fact is being rewritten) but is currently the state of the art in terms of running arbitrary code as WASM.
We now whole-heartedly support accepting WASI workloads. We now also support specifying arbitrary input and output volumes for WASM jobs as well. Files can be opened from the WASI interface and read or written as expected.
WASM jobs are therefore now capable of reading data from input volumes, writing logs via STDOUT and STDERR, returning an exit code, and writing arbitrary files to output volumes.
There is no support for listing directory contents yet - jobs have to know what files they are expecting. In a future commit, we could add support for specifying input arguments to the program to work around this. Full directory listing support will require upgrades to the wazero runtime.
With latest Bacalhau main checked out, you can now:
- Add inputs to IPFS:
ipfs add -r ~/bacalhau/testdata/wasm/csv/inputs
- Run WASM:
bacalhau wasm run ~/bacalhau/testdata/wasm/csv/main.wasm _start -v QmeVbdq3vx4P8LJMWmjPaYAzxjMg6nNNJdw9rYFrMvqqy1:inputs
where QmeVbdq3vx4P8LJMWmjPaYAzxjMg6nNNJdw9rYFrMvqqy1 is the CID from step 1.
We had a lot of regressions on the Examples pipeline as a result of some (great!) UX changes. These are now green again thanks to a heroic effort from Phil!
He also fixed a URL downloading bug, sped up the tests, and landed [docs for Stable Diffusion]!
CoD in space!! 🚀 😁
We're now referring to the DAG work as job pipelining. And Enrico has made excellent progress on the design doc!
What's more, we've settled on starting work to prototype Bacalhau-Airflow integration! I'm really excited about this as it will open Bacalhau up to a community of Airflow pipeline authors who -- with minimal effort -- will be able to port their existing pipelines to run on our open, distributed network!
Enrico is also hacking away at getting our OVH machines online, and has updated our Service Provider docs. If you're a Service Provider, interested in becoming a Compute Provider, please give them a try and give us feedback on #bacalhau on the Filecoin Slack!!
Next week is our final push on UX polish and examples before the Lisbon CoD summit -- so it will be all hands on deck for that.
There might also be some cool new demos ready in time for that ;-)