RuntimeError when generating cloud init config for Packer (GCPCluster)

[alexwalterbos]

Describe the issue:

I'm trying to build a custom OS image for my GCP cluster with packer. I'm using these steps: https://cloudprovider.dask.org/en/latest/packer.html

The util get_cloud_init() fails, seemingly on some underlying async logic.

Minimal Complete Verifiable Example:

from dask_cloudprovider.gcp import GCPCluster

if __name__ == '__main__':
    cic = GCPCluster.get_cloud_init(
        projectid='valid-project-id-that-works-for-cluster-creation',
        zone='europe-xxxx-x',
        machine_type='c2-standard-8', # but occurs with other machine types as well
        env_vars={
            "EXTRA_PIP_PACKAGES": "joblib==1.3.2 sqlalchemy==2.0.13 scikit-learn==1.3.2 pandas==2.1.4 numpy==1.26.2 nltk==3.6.7"},
    )
    print(cic)

Anything else we need to know?:
Error traceback:

/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/bin/python3 /home/alexwalterbos/repos/magnet.me/services/recommender/similarity/learn/esco/classifiers/get_cloud_config.py
Traceback (most recent call last):
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/learn/esco/classifiers/get_cloud_config.py", line 6, in <module>
    cic = GCPCluster.get_cloud_init(
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 365, in get_cloud_init
    cluster = cls(*args, asynchronous=True, **kwargs)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/dask_cloudprovider/gcp/instances.py", line 634, in __init__
    super().__init__(debug=debug, **kwargs)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 297, in __init__
    super().__init__(**kwargs, security=self.security)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/distributed/deploy/spec.py", line 275, in __init__
    super().__init__(
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/distributed/deploy/cluster.py", line 70, in __init__
    self._loop_runner = LoopRunner(loop=loop, asynchronous=asynchronous)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/distributed/utils.py", line 533, in __init__
    raise RuntimeError(
RuntimeError: Constructing LoopRunner(loop=loop) without a running loop is not supported

Environment:

• Dask version: '2023.12.1'
• Python version: 3.10.13
• Operating System: Fedora Linux 6.6.8-200.fc39.x86_64
• Install method (conda, pip, source): pip

Thanks in advance for any assistance you can provide.

[dbalabka]

Here is a hack on how to workaround the issue:

from dask_cloudprovider.gcp import GCPCluster
import asyncio

try:
    asyncio.get_running_loop()
    # In case of event loop is already running, simply call the function
    cic = GCPCluster.get_cloud_init(**cluster_params)
except RuntimeError as e:
    async def get_cloud_init():
        """Hack to fix https://github.com/dask/dask-cloudprovider/issues/421 """
        return GCPCluster.get_cloud_init(**cluster_params)

    cic = asyncio.get_event_loop().run_until_complete(get_cloud_init())

print(cic)

The problem is still reproducible. However, we do not notice it inside Jupyter because it wraps all cells into an event loop.

[dbalabka]

Unfortunately, documenation does not mention that get_cloud_init should be run inside the event loop:
https://github.com/dask/dask-cloudprovider/blob/c6eb82639fb87c0d4a348460683e9a201f4c45e5/doc/source/packer.rst#ec2cluster-with-cloud-init

while tests are marked as async:

dask-cloudprovider/dask_cloudprovider/gcp/tests/test_gcp.py

Line 56 in c6eb826

@pytest.mark.asyncio