Update deployment to use Dask Kubernetes Operator

[matthewfeickert]

So it feels like the time to finally retire the classic KubeCluster implementation along with the HelmCluster.

I suspect that the update to dask-kubernetes is the reason. Maybe some JupyterLab integration would need to be updated accordingly, or perhaps some other backend thing, but for now I think it would be good to revert the change.

Originally posted by @alexander-held in usatlas/analysisbase-dask#49 (comment)

The Dask Kubernetes Operator has been the standard for deploying Dask on Kubernetes for over a year now. So it feels like the time to finally retire the classic KubeCluster implementation along with the HelmCluster.

https://github.com/dask/dask-kubernetes/releases/tag/2024.5.0

This would be useful to switch to as otherwise Dask will be stuck on the 2024.4.x releases.

@ivukotic is this possible?

[matthewfeickert]

You can see our dask configuration in the derived AB image or in Jupyter directly. Our clusters are deployed using Dask Operators. (We are testing a Dask Gateway with authentication and everything but don’t want to change anything till the DESY meetings. — @ivukotic

(From IRIS-HEP Slack)

@ivukotic Sorry, I'm not experience here, so can you point out what section of https://github.com/usatlas/analysisbase-dask-uc/blob/ccd0a8000c6ee3d82b53f3e022d81c4f4bc8d34f/dask_config.yaml is this specified in (or what section of the Dask Kubernetes Operator docs I should be looking at)? Also, if the deployment is already done with Dask Operators, why did @alexander-held encounter the issue from usatlas/analysisbase-dask#48?

(We are testing a Dask Gateway with authentication and everything but don’t want to change anything till the DESY meetings).

👍 Nice, thank you!

[matthewfeickert]

@LincolnBryant @fengpinghu can you also take a look at this?

[fengpinghu]

It looks it's indeed using the classic KubeCluster because in the configuration file KubeCluster users worker template and scheduler template to create pods directly. We will look into either dask-operator or dask-gateway integration.

[matthewfeickert]

We will look into either dask-operator or dask-gateway integration.

Great! Thanks very much, @fengpinghu!