HPC Cluster Support (Ray)

[trahloff]

What would you like to be added: Support for using JuiceFS with HPC clusters like Ray.

Why is this needed:
JuiceFS is an amazing tool that makes it extremely easy to work with data that is stored on S3 but needs to be instantly available for high parallel i/o. In our case, this use case is "machine learning on satellite images".

Having this integrated into k8s via CSI is great but constraints the usecases to workloads that fit well into a k8s environment. Because of our extremely high ressource-requirements, we and many other companies that are running data processing pipelines rely on compute clusters like Ray.
Do you see any possibility in integrating JuiceFS natively into Ray?

[SandyXSD]

That sounds interesting, but currently we still lack native Python APIs. So for now, you have to pick one of the three interfaces (mount as a network file system, S3 gateway, HDFS) provided by JuiceFS to interact with Ray.

[trahloff]

Hi @SandyXSD, thanks for the quick response! Would it work out-of-the-box if Ray clusters would support CSI?

[SandyXSD]

Yes, it should be.

[one-matrix]

hi.@trahloff @SandyXSD ,Ray can run on k8s like kuberay. and JuiceFS support CSI in k8s. I think this is a feasible solution but I don't see anyone doing that.I'm trying to go in that way.

[trahloff]

Hi @one-matrix. Yep, it would work via k8s. We explored this option but decided against it because it would introduce unneeded complexity, just for supporting JuiceFS.

Most stacks I know use Ray directly with AWS Spot Instances (or equivalent from other hyperscalers) because introducing a middle-layer like k8s or ECS does not come with huge benefits and most stacks are better of giving Ray the responsibility of scheduling the underlying infrastructure.

But if it makes sense in your case to run Ray on k8s, by all means, go for it.
Thanks for updating this ticket. You are completely right, this setup solves the requirement of integration JuiceFS with Ray 👍

[davies]

In the case of AWS Spot Instances, can you setup JuiceFS within the init-script?