AKS: add cluster publick8s and migrate prodpublick8s public services on it

[dduportal]

This issue tracks the work for spawning a new "public" AKS cluster for production to replace the former prodpublick8s.

Goals:

• Get rid of the network overlap from the current AKS prodpublick8s
• Ensure that the cluster is managed as code
• Support IPv6 for services hosted in this cluster (requires NOT using CNI for this cluster, forbidding us to use Windows node pools)
  • Ref. https://learn.microsoft.com/en-us/azure/aks/configure-kubenet-dual-stack?tabs=azure-cli%2Ckubectl
• Ensure a new "home" for (Re) Introduce an artifact caching proxy for ci.jenkins.io #2752 in Azure environment

This issue is the "twin" of #2844 but for public network.

---

Some notes:

• Node pools:
  • No Windows node pool is expected (only used by release.ci.jenkins.io which will be migrated to privatek8s)
  • No Highmem node pool is expected (only used by release.ci.jenkins.io which will be migrated to privatek8s - Only reference to highmem string in the context of Kubernete in https://github.com/search?q=org%3Ajenkins-infra+highmem&type=code is https://github.com/jenkins-infra/release/blob/902df23d5657c074f184d8e08d1d00d7e3e67c95/PodTemplates.d/release-linux.yaml#L44)
  • The current agentpool for prodpublick8s should have an equivalent (in term of VM size, disk size and autoscaling limits) in the new privatek8s
  • A "default system node pool" is expected, like for privatek8s, as a good AKS practise (ref. https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli and https://learn.microsoft.com/en-us/azure/aks/use-multiple-node-pools)
• The Add a VPN for public network #3306 is NOT needed (we can reach the private-ingress endpoint using the private's VPN network peering to public)
• We'll have to decide which can of "egress" (attribute outboundType) type to use: https://learn.microsoft.com/en-us/azure/aks/egress-outboundtype
• We'll have to ensure that the CSI storage classe are set to retain for the non-default, and that any "custom" CSI classe is installed
• We'll have to install in this cluster the following (usual) side services:
  • datadog
  • certmanager+acme
  • Nginx (private) ingress controller (with an internal load balancer that should be reachable from the peered private network), and should be the default
  • Nginx (public) ingress controller (with a public LB and public IP)
  • Falco (to ensure some policies are enforced on workloads)
• We'll have to migrate the following services from prodpublick8s to this cluster:
  • plugin-site (plugins.jenkins.?io)
  • plugin-site-issues
  • reports (reports.jenkins.io)
  • jenkinsio (⚠️ when changing the DNS, we'll have to watch fastly) (origin.jenkins.io)
  • javadoc (javadoc.jenkins.io)
  • accountapp (accounts.jenkins.io)
  • ldap (ldap.jenkins.io with its own public LB)
  • uplink (uplink.jenkins.io)
  • mirrorbits (get.jenkins.io)
  • keycloak
  • incrementals-publisher
  • jenkins-weekly
  • jenkinsisthewayio-redirect
  • artifact-caching-proxy
  • plugin-health-scoring
  • wiki
  • rating

Tasks

Preview Give feedback

No tasks being tracked yet.

[lemeurherve]

With this migration we'll be able to close #3209

[dduportal]

Putting on hold: #2844 tracks the migration of release.ci from prodpublick8s to privatek8s, before proceeding forward here.

[lemeurherve]

Cleanup of unused DNS records found while working on this issue

A records pointing to 52.167.253.43 (prodpublick8s public IP)

• public.aks.jenkins.io
• jenkins-ci.org (apex record), to be imported in terraform and changed to 20.119.232.75 (publick8s public IP)
  • Import: fix(dns) import apex record for jenkins-ci.org azure-net#110
  • IP change: feat(dns): change jenkins-ci.org apex to publick8s azure-net#111

CNAME records pointing to publick.aks.jenkins.io (ie prodpublick8s cluster):

• archives.azure.jenkins.io
• mirror.azure.jenkins.io^*
• polls.jenkins.io
• beta.accounts.jenkins.io
• customize.jenkins.io

A records pointing to 10.0.2.5 (prodpublick8s private IP)

• private.aks.jenkins.io

CNAME records pointing to private.aks.jenkins.io (ie prodpublick8s cluster):

• release.pkg.jenkins.io
• admin.polls.jenkins.io
• release.repo.jenkins.io

Miscellaneous

• archives.azure.jenkins.io
• mirrors.azure.jenkins.io
• mirrors2.jenkins-ci.org

^*: need additional cleanup in:

• mirrorbits helm chart README
• runbook
• datadog checks.

[lemeurherve]

As we've noticed quite a lot of remaining requests still send to mirrorbits on prodpublick8s, we'll postpone the cluster deletion to next week, and @dduportal will see for the publication of a blogpost indicating the migration of this service to the new cluster.

[dduportal]

Namespaces removal: @lemeurherve and I paired and removed the following namespaces from prodpublick8s:

• datadog (causing cleanup: ensure no more metrics from prodpublick8s are used for monitors datadog#193 😅)
• cert-manager
• private-nginx-ingress.

Remaining namespaces are required until #3351 (comment) is fixed.

[dduportal]

As we've noticed quite a lot of remaining requests still send to mirrorbits on prodpublick8s, we'll postpone the cluster deletion to next week, and @dduportal will see for the publication of a blogpost indicating the migration of this service to the new cluster.

As discussed with the last infrastructure meeting:

• We are planning to decommission the former prodpublick8s cluster the Tuesday 27 June 2023, including the former mirror service and the former IPv4
• Community thread in: https://community.jenkins.io/t/new-public-ipv4-for-jenkins-mirrors/8137
• Jenkins Blog post:
  • PR: Add a blog post for Jenkins Mirrors IP change jenkins.io#6483
  • Direct link:

[dduportal]

Update:

• Cluster prodpublick8s destroyed with its resource group
• IPv6 problem with “get.jenkins.io” #3639 should be fixed

[lemeurherve]

Additional monitors added: jenkins-infra/datadog#195

Potential improvements for later:

• use a PV/PVC instead of azurefile for plugin-site storage
• evaluate if plugin-site needs the same GitHub and Jira credentials as plugin-site-api (optional)
• create an issue for a mechanism to display a maintenance page on services while operating on them
• activate the "secure transfer required" for javadoc storage account
• add cluster outbound IPs as terraform output

After 3 years and 27 days of good and faithful service, prodpublick8s is not anymore, closing this issue 🤗