Skip to content

ECSCluster/FargateCluster ignores cluster_name when cluster_arn is provided #472

@dicknetherlands

Description

@dicknetherlands

I am using a single existing ECS cluster to launch multiple Dask clusters simultaneously, using existing security groups and roles rather than letting Dask CloudProvider create the default ones. However I am still allowing Dask CloudProvider to auto-generate task definitions.

Dask CloudProvider prefixes the internal variable self.cluster_name to all the generated task definitions. For auto-generated clusters this means that every cluster and its associated task definition set get a unique prefix (using cluster_name_template), so they are easy to distinguish at a glance, but for pre-existing ECS clusters they all get prefixed the same using the name of the ECS cluster so it is much harder to tell which task is running on which Dask cluster within the single parent ECS cluster.

Each of those Dask cluster's task definitions and associated auto-generated entities should have a name/prefix to uniquely identify them even when sharing the same parent ECS cluster. This makes debugging much easier by being able to quickly visually associate task definitions with the correct launch of the Dask cluster.

The easiest way to do this would be to have ECSCluster honour and save the cluster_name parameter even when cluster_arn is also provided. Currently it ignores it completely and always uses the name of the actual ECS cluster. By accepting it as a constructor parameter and saving it, the rest of the code then automatically does the right thing and prefixes it to all auto-generated entities instead of the actual name of the existing ECS cluster.

Example code (assertion should pass, but it doesn't):

cluster = ECSCluster(cluster_name="xyx", cluster_arn="arn:aws:....yourarnhere....")
assert cluster.cluster_name == "xyz"

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions