Weird behaviour creating a new cluster on stackit

[h3adex]

Issue description

While setting up a Constellation cluster on Stackit, I noticed something unexpected with the Kubernetes version. I followed the getting started documentation and used the command constellation config generate stackit.

However, I was surprised to see that I received Kubernetes version 1.29, even though I expected version 1.30, which is the latest supported version. The logs also presented a few puzzling messages:

Plan: 2 to add, 0 to change, 0 to destroy.
Changes of Constellation cloud resources are required by applying an updated Terraform template.
Do you want to apply these Terraform changes? [y/n]: y
Applying Terraform changes  
Infrastructure migrations applied successfully and output written to: constellation-state.yaml
A backup of the pre-upgrade state has been written to: constellation-upgrade/upgrade-20241016103833-bf6cf8ae/terraform-backup
invalid upgrade for cilium: upgrading from v1.15.5-edg.1 to v1.15.5-edg.1 is not a valid upgrade: current version newer than or equal to new version
invalid upgrade for constellation-services: upgrading from v2.18.0 to v2.18.0 is not a valid upgrade: current version newer than or equal to new version
invalid upgrade for cert-manager: upgrading from v1.15.0 to v1.15.0 is not a valid upgrade: current version newer than or equal to new version
invalid upgrade for constellation-operators: upgrading from v2.18.0 to v2.18.0 is not a valid upgrade: current version newer than or equal to new version
invalid upgrade for constellation-csi: upgrading from v2.18.0 to v2.18.0 is not a valid upgrade: current version newer than or equal to new version
invalid upgrade for yawol: upgrading from v0.0.0 to v0.0.0 is not a valid upgrade: current version newer than or equal to new version
Upgrading Kubernetes components   
Successfully upgraded Constellation services.
skipping Kubernetes upgrade: upgrading from v1.30.4 to v1.29.8 is not a valid upgrade: current version newer than or equal to new version

(skipping Kubernetes upgrade: upgrading from v1.30.4 to v1.29.8 is not a valid upgrade: current version newer than or equal to new version)

It seems there’s a misunderstanding in the version comparison—1.29.8 is not newer or equal to 1.30.4. Could you please look into these inconsistencies when you have a chance?

constellation upgrade check also throws an error. But this might be an expected behavior since Image upgrades are not supported for provider OpenStack. Imo this should also be added to the docs: https://docs.edgeless.systems/constellation/workflows/upgrade. Just FYI

2024/10/16 11:09:55 [DEBUG] GET
https://releases.hashicorp.com/terraform/index.json
Error: getting cluster versions: expected exactly one condition, got 0

Steps to reproduce the behavior

Follow the steps on creating a cluster
https://docs.edgeless.systems/constellation/next/getting-started/first-steps

Version

Version: v2.18.0 (Enterprise build; see documentation for license agreement)
GitCommit: 57af2d3
GitTreeState: clean
BuildDate: 2024-09-26T10:10:47
GoVersion: go1.22.7 X:nocoverageredesign
Compiler: bazel/gc
Platform: darwin/arm64

Constellation Config

version: v4 # Schema version of this configuration file.
image: v2.18.0 # Machine image version used to create Constellation nodes.
name: constell # Name of the cluster.
kubernetesVersion: v1.29.8 # Kubernetes version to be installed into the cluster.
microserviceVersion: v2.18.0 # Microservice version to be installed into the cluster. Defaults to the version of the CLI.
debugCluster: false # DON'T USE IN PRODUCTION: enable debug mode and use debug images.
customEndpoint: "" # Optional custom endpoint (DNS name) for the Constellation API server.
internalLoadBalancer: false # Flag to enable/disable the internal load balancer. If enabled, the Constellation is only accessible from within the VPC.
serviceCIDR: 10.96.0.0/12 # The Kubernetes Service CIDR to be used for the cluster. This value will only be used during the first initialization of the Constellation.
# Additional tags that are applied to created resources.
tags: {}
# Supported cloud providers and their specific configurations.
provider:
    # Configuration for OpenStack as provider.
    openstack:
        cloud: stackit # OpenStack cloud name to select from "clouds.yaml". Only required if config file for OpenStack is used. Fallback authentication uses environment variables. For details see: https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html.
        cloudsYAMLPath: "" # Path to OpenStack "clouds.yaml" file. Only required if automatic detection fails.
        availabilityZone: "eu01-1" # Availability zone to place the VMs in. For details see: https://docs.openstack.org/nova/latest/admin/availability-zones.html
        floatingIPPoolID: 970ace5c-458f-484a-a660-0903bcfd91ad # Floating IP pool to use for the VMs. For details see: https://docs.openstack.org/ocata/user-guide/cli-manage-ip-addresses.html
        stackitProjectID: "REMOVED" # STACKITProjectID is the ID of the STACKIT project where a user resides.
        regionName: RegionOne # description: |
        deployYawolLoadBalancer: true # Deploy Yawol loadbalancer. For details see: https://github.com/stackitcloud/yawol
        yawolImageID: bcd6c13e-75d1-4c3f-bf0f-8f83580cc1be # OpenStack OS image used by the yawollet. For details see: https://github.com/stackitcloud/yawol
        yawolFlavorID: 3b11b27e-6c73-470d-b595-1d85b95a8cdf # OpenStack flavor id used for yawollets. For details see: https://github.com/stackitcloud/yawol
        deployCSIDriver: true # Deploy Cinder CSI driver with on-node encryption. For details see: https://docs.edgeless.systems/constellation/architecture/encrypted-storage
# Node groups to be created in the cluster.
nodeGroups:
    control_plane_default:
        role: control-plane # Role of the nodes in this group. Valid values are "control-plane" and "worker".
        zone: "eu01-1" # Availability zone to place the VMs in.
        instanceType: m1a.4cd # VM instance type to use for the nodes.
        stateDiskSizeGB: 30 # Size (in GB) of a node's disk to store the non-volatile state.
        stateDiskType: storage_premium_perf6 # Type of a node's state disk. The type influences boot time and I/O performance.
        initialCount: 3 # Number of nodes to be initially created.
    worker_default:
        role: worker # Role of the nodes in this group. Valid values are "control-plane" and "worker".
        zone: "eu01-1" # Availability zone to place the VMs in.
        instanceType: m1a.4cd # VM instance type to use for the nodes.
        stateDiskSizeGB: 30 # Size (in GB) of a node's disk to store the non-volatile state.
        stateDiskType: storage_premium_perf6 # Type of a node's state disk. The type influences boot time and I/O performance.
        initialCount: 1 # Number of nodes to be initially created.

[3u13r]

Hello,

thanks for the detailed issue.

I just created a Constellation on STACKIT and received the Kubernetes version I specified in the config.

We use the apply command both for upgrades and first creations (similar to what Terraform does). Therefore, from your first output, especially "Plan: 2 to add, 0 to change, 0 to destroy." and "skipping Kubernetes upgrade: upgrading from v1.30.4 to v1.29.8 is not a valid upgrade" my guess would be that the Constellation with Kubernetes version 1.30 already existed before the constellation apply.

You can compare the uid in constellation-state.yaml with the names of the resources in the portal / openStack CLI.
Furthermore, when creating a cluster for the first time, you always should receive the following output:

euler@work:~/constellationReleaseTest/2.18.0$ ./constellation apply
Using marketplace image billing.
Checking for infrastructure changes
The following Constellation cluster will be created:
  3 control-plane nodes of type m1a.4cd will be created.
  1 worker node of type m1a.4cd will be created.
Do you want to create this cluster? [y/n]:

Could you maybe re-test what you did in a new folder containing only the constellation binary?

[h3adex]

I just retested the process, and here's what I found: creating a cluster with Kubernetes versions 1.29.8 and 1.30.4 works as expected. I'm also able to reapply new node pools without any issues. Additionally, the Kubernetes version specified in my configuration is applied correctly.

The incorrect log message regarding cluster upgrades has disappeared when I create a 1.29.8 cluster and, for example, apply a new node pool. Has there been any update or change made to your API recently?
Before: skipping Kubernetes upgrade: upgrading from v1.30.4 to v1.29.8 is not a valid upgrade: current version newer than or equal to new version
Now: skipping Kubernetes upgrade: upgrading from v1.29.8 to v1.29.8 is not a valid upgrade: current version newer than or equal to new version

However, problems I encountered which are not a showstopper:

• Why does the default configuration generate with Kubernetes version 1.29.8 instead of the newer version 1.30.4?
• Running a constellation upgrade check results in an error, but enabling debug logs hasn't provided any insights into why this error persists.

[3u13r]

Has there been any update or change made to your API recently?

What API are you referring to? The Kubernetes version comparison happens only between the cluster and the cli.

Why does the default configuration generate with Kubernetes version 1.29.8 instead of the newer version 1.30.4?

We always have the middle of the 3 supported Kubernetes version as default. All the major CSPs also have a conservative default version (GKE is currently 1.30, EKS 1.30 and Azure 1.29). We will bump the default version as soon as #3396 is merged. Until then, you can use K8s version 1.30 by simply setting it in the config.

Running a constellation upgrade check results in an error, but enabling debug logs hasn't provided any insights into why this error persists.

We should catch the error earlier in the CLI, but upgrades on STACKIT are not supported right now. We will add a message in future versions explaining this. The reason it is not implemented is that on the other CSPs we use ScaleSets/InstanceGroups. This is currently not available on STACKIT, therefore we would need to implement a different upgrade logic than with the other CSPs.

[h3adex]

What API are you referring to? The Kubernetes version comparison happens only between the cluster and the CLI.

I haven’t changed anything on my end. It's the same Constellation CLI and Kubernetes version. However, the log message is now correct. I thought perhaps you had updated some information on your backend that the CLI uses to determine whether components need updating.

We always have the middle of the 3 supported Kubernetes versions as default. All the major CSPs also have a conservative default version (GKE is currently 1.30, EKS 1.30, and Azure 1.29). We will bump the default version as soon as #3396 is merged. Until then, you can use K8s version 1.30 by simply setting it in the config.

Thank you for clarifying. This information was not apparent to me, possibly because I missed it in the documentation. It makes sense that 1.29 is the default version, though my initial expectation was to have the latest version by default.

We should catch the error earlier in the CLI, but upgrades on STACKIT are not supported right now. We will add a message in future versions explaining this. The reason it is not implemented is that on the other CSPs we use ScaleSets/InstanceGroups. This is currently not available on STACKIT, therefore we would need to implement a different upgrade logic than with the other CSPs.

OK! Thank you for the information!

[h3adex]

I guess we can close this issue now?