rebuild the AKS build cluster

[upodroid]

Our AKS build cluster hasn't been functional for quite some time. https://prow.k8s.io/?cluster=k8s-infra-aks-admin

So I rebuilt it to support some of our more modern requirements such as dualstack networking, mandatory workload identity, and being able to select pod sizes(you label your pod with special labels, and we mutate the pod to have specific cpu/memory)

So I rebuilt the cluster:

• Connected it to ArgoCD
• Dualstack Cluster, xref: Rebuild our Prow Build Clusters to support dualstack networking #8904
• Has both amd64 and arm64 nodepool
• The cluster name in prow will change from k8s-infra-aks-admin to k8s-infra-prow-aks-build
• Kyverno is deployed to mutate the pods and insert GCP creds required by prow, also to deploy this: Document CI cluster selection, CPU : RAM ratio / machine types, and general recommendations specific to prow.k8s.io test-infra#34139 (comment)

The cluster is already up and running, and the infra changes have been applied.

/cc @ameukam @hakman @xmudrii @GenPage

[upodroid]

bump

[xmudrii]

/lgtm
/approve
/hold
unhold when ready

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• infra/azure/OWNERS [upodroid,xmudrii]
• infra/gcp/terraform/k8s-infra-prow-build/OWNERS [upodroid,xmudrii]
• kubernetes/OWNERS [upodroid,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[upodroid]

atlantis plan

[k8s-infra-ci-robot]

Ran Plan for 2 projects:

1. dir: infra/azure/terraform/k8s-infra-prow-build workspace: default
2. dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

---

1. dir: infra/azure/terraform/k8s-infra-prow-build workspace: default

Plan Error

Show Output

running 'sh -c' '/usr/local/bin/terraform init -input=false -upgrade' in '/atlantis/repos/kubernetes/k8s.io/8911/default/infra/azure/terraform/k8s-infra-prow-build': exit status 1
Initializing the backend...
Upgrading modules...
Downloading registry.terraform.io/claranet/regions/azurerm 7.4.0 for azure_region...
- azure_region in .terraform/modules/azure_region
Downloading registry.terraform.io/Azure/naming/azurerm 0.4.1 for naming...
- naming in .terraform/modules/naming
Downloading registry.terraform.io/Azure/avm-ptn-network-private-link-private-dns-zones/azurerm 0.23.0 for private_dns_zones...
- private_dns_zones in .terraform/modules/private_dns_zones
Downloading registry.terraform.io/Azure/avm-utl-interfaces/azure 0.5.0 for private_dns_zones.avm_interfaces...
- private_dns_zones.avm_interfaces in .terraform/modules/private_dns_zones.avm_interfaces
Downloading registry.terraform.io/Azure/avm-res-network-privatednszone/azurerm 0.4.3 for private_dns_zones.avm_res_network_privatednszone...
- private_dns_zones.avm_res_network_privatednszone in .terraform/modules/private_dns_zones.avm_res_network_privatednszone
- private_dns_zones.avm_res_network_privatednszone.a_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_a_record
- private_dns_zones.avm_res_network_privatednszone.aaaa_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_aaaa_record
Downloading registry.terraform.io/Azure/avm-utl-interfaces/azure 0.2.0 for private_dns_zones.avm_res_network_privatednszone.avm_interfaces...
- private_dns_zones.avm_res_network_privatednszone.avm_interfaces in .terraform/modules/private_dns_zones.avm_res_network_privatednszone.avm_interfaces
- private_dns_zones.avm_res_network_privatednszone.cname_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_cname_record
- private_dns_zones.avm_res_network_privatednszone.mx_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_mx_record
- private_dns_zones.avm_res_network_privatednszone.ptr_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_ptr_record
- private_dns_zones.avm_res_network_privatednszone.srv_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_srv_record
- private_dns_zones.avm_res_network_privatednszone.txt_record in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_txt_record
- private_dns_zones.avm_res_network_privatednszone.virtual_network_links in .terraform/modules/private_dns_zones.avm_res_network_privatednszone/modules/private_dns_virtual_network_link
Downloading registry.terraform.io/Azure/avm-utl-regions/azurerm 0.9.2 for private_dns_zones.regions...
- private_dns_zones.regions in .terraform/modules/private_dns_zones.regions
- private_dns_zones.regions.cached_data in .terraform/modules/private_dns_zones.regions/modules/cached-data
Downloading registry.terraform.io/Azure/aks/azurerm 10.0.0 for prow_build...
- prow_build in .terraform/modules/prow_build/v4
Downloading registry.terraform.io/Azure/avm-res-network-virtualnetwork/azurerm 0.16.0 for prow_network...
- prow_network in .terraform/modules/prow_network
- prow_network.peering in .terraform/modules/prow_network/modules/peering
- prow_network.subnet in .terraform/modules/prow_network/modules/subnet
╷
│ Error: unable to build authorizer for Resource Manager API: could not configure AzureCli Authorizer: could not parse Azure CLI version: launching Azure CLI: exec: "az": executable file not found in $PATH
│ 
│ 
╵

---

2. dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # google_iam_workload_identity_pool.aks_cluster will be created
+ resource "google_iam_workload_identity_pool" "aks_cluster" {
      + description               = "Identity pool for CI on Azure using AKS clusters"
      + display_name              = "AKS Prow Cluster"
      + id                        = (known after apply)
      + name                      = (known after apply)
      + project                   = "k8s-infra-prow-build"
      + state                     = (known after apply)
      + workload_identity_pool_id = "prow-aks"
    }

  # google_iam_workload_identity_pool_provider.aks_cluster will be created
+ resource "google_iam_workload_identity_pool_provider" "aks_cluster" {
      + attribute_mapping                  = {
          + "google.subject" = "assertion.sub"
        }
      + description                        = "Identity pool for CI on Azure using AKS clusters"
      + display_name                       = "AKS OIDC provider"
      + id                                 = (known after apply)
      + name                               = (known after apply)
      + project                            = "k8s-infra-prow-build"
      + state                              = (known after apply)
      + workload_identity_pool_id          = "prow-aks"
      + workload_identity_pool_provider_id = "oidc"

      + oidc {
          + allowed_audiences = [
              + "sts.googleapis.com",
            ]
          + issuer_uri        = "https://eastus2.oic.prod-aks.azure.com/d1aa7522-0959-442e-80ee-8c4f7fb4c184/85d5aa19-bc3c-4cdb-bc17-0cf8703cfa3f"
        }
    }

  # google_vmwareengine_network_peering.gvce_peering will be updated in-place
~ resource "google_vmwareengine_network_peering" "gvce_peering" {
      ~ export_custom_routes_with_public_ip = false -> true
        id                                  = "projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project"
      ~ import_custom_routes_with_public_ip = false -> true
        name                                = "peer-with-gcve-project"
        # (13 unchanged attributes hidden)
    }

  # module.workload_identity_service_accounts["kubernetes-external-secrets"].data.google_iam_policy.workload_identity will be read during apply
  # (config refers to values not yet known)
 <= data "google_iam_policy" "workload_identity" {
      + id          = (known after apply)
      + policy_data = (known after apply)

      + binding {
          + members = [
              + "principalSet://iam.googleapis.com/projects/773781448124/locations/global/workloadIdentityPools/prow-eks/*",
              + "serviceAccount:k8s-infra-prow-build.svc.id.goog[kubernetes-external-secrets/kubernetes-external-secrets]",
              + (known after apply),
            ]
          + role    = "roles/iam.workloadIdentityUser"
        }
    }

  # module.workload_identity_service_accounts["kubernetes-external-secrets"].google_service_account_iam_policy.serviceaccount_iam will be updated in-place
~ resource "google_service_account_iam_policy" "serviceaccount_iam" {
        id                 = "projects/k8s-infra-prow-build/serviceAccounts/kubernetes-external-secrets@k8s-infra-prow-build.iam.gserviceaccount.com"
      ~ policy_data        = jsonencode(
            {
              - bindings = [
                  - {
                      - members = [
                          - "principalSet://iam.googleapis.com/projects/773781448124/locations/global/workloadIdentityPools/prow-eks/*",
                          - "serviceAccount:k8s-infra-prow-build.svc.id.goog[kubernetes-external-secrets/kubernetes-external-secrets]",
                        ]
                      - role    = "roles/iam.workloadIdentityUser"
                    },
                ]
            }
        ) -> (known after apply)
        # (2 unchanged attributes hidden)
    }

Plan: 2 to add, 2 to change, 0 to destroy.

• ▶️ To apply this plan, comment:

  atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build

• 🚮 To delete this plan and lock, click here
• 🔁 To plan this project again, comment:

  atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 2 to add, 2 to change, 0 to destroy.

---

Plan Summary

2 projects, 1 with changes, 0 with no changes, 1 failed

• ⏩ To apply all unapplied plans from this Pull Request, comment:

  atlantis apply

• 🚮 To delete all plans and locks from this Pull Request, comment:

  atlantis unlock

[upodroid]

atlantis apply

[k8s-infra-ci-robot]

Ran Apply for dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output

google_iam_workload_identity_pool.aks_cluster: Creating...
google_vmwareengine_network_peering.gvce_peering: Modifying... [id=projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project]
google_vmwareengine_network_peering.gvce_peering: Still modifying... [id=projects/k8s-infra-prow-build/locations...networkPeerings/peer-with-gcve-project, 00m10s elapsed]
google_iam_workload_identity_pool.aks_cluster: Still creating... [00m10s elapsed]
google_iam_workload_identity_pool.aks_cluster: Creation complete after 10s [id=projects/k8s-infra-prow-build/locations/global/workloadIdentityPools/prow-aks]
module.workload_identity_service_accounts["kubernetes-external-secrets"].data.google_iam_policy.workload_identity: Reading...
google_iam_workload_identity_pool_provider.aks_cluster: Creating...
module.workload_identity_service_accounts["kubernetes-external-secrets"].data.google_iam_policy.workload_identity: Read complete after 0s [id=1316087891]
module.workload_identity_service_accounts["kubernetes-external-secrets"].google_service_account_iam_policy.serviceaccount_iam: Modifying... [id=projects/k8s-infra-prow-build/serviceAccounts/kubernetes-external-secrets@k8s-infra-prow-build.iam.gserviceaccount.com]
module.workload_identity_service_accounts["kubernetes-external-secrets"].google_service_account_iam_policy.serviceaccount_iam: Modifications complete after 1s [id=projects/k8s-infra-prow-build/serviceAccounts/kubernetes-external-secrets@k8s-infra-prow-build.iam.gserviceaccount.com]
google_vmwareengine_network_peering.gvce_peering: Modifications complete after 11s [id=projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project]
google_iam_workload_identity_pool_provider.aks_cluster: Still creating... [00m10s elapsed]
google_iam_workload_identity_pool_provider.aks_cluster: Creation complete after 11s [id=projects/k8s-infra-prow-build/locations/global/workloadIdentityPools/prow-aks/providers/oidc]

Apply complete! Resources: 2 added, 2 changed, 0 destroyed.

[k8s-infra-ci-robot]

Locks and plans deleted for the projects and workspaces modified in this pull request:

• dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default