deepCopy of values to linkerd-config and Helm post-renderer scripts

[gaborho]

Dear linkerd Community, I use Helm chart to deploy linkerd at our company. Because of a regulated environment, EKS worker nodes are not able to pull Docker container images from external container registries, they must pull from ECR.

In the Ansible role I have some tasks that:

1. render chart locally and determine images to be re-tagged
2. login to ECR, create ECR repos if required
3. pull the images through company Docker proxy
4. re-tag images (setting target registry hostname to ECR)
5. push images to ECR
6. prepare kustomization.yaml for kustomize
7. call kustomize as Helm post-rendering shell script which will then rewrite the images in the final template before they are installed by Helm.

For linkerd-config Configmap there is a deepCopy for the Values.

This unfortunately leads to the problem that initially I have original ghcr.io host in image definitions for proxy-init and proxy, and not the correct ECR ones, since kustomize modify to that before "helm install" in the post-renderer.

Therefore when I do kubectl rollout restart for deployments in the cluster (other than linkerd deployments), they fail to pull the image, even though linkerd is running properly (meshed).
I workarounded that by calling additionally after the initial "helm install" a "helm upgrade" where I set individually every image name of linkerd:
helm upgrade linkerd2 linkerd-stable/linkerd2 -n {{ linkerd_namespace }} --reuse-values \ --set global.proxy.image.name={{ ecr }}/linkerd/proxy \ --set global.proxyInit.image.name={{ ecr }}/linkerd/proxy-init \ --set controllerImage={{ ecr }}/linkerd/controller \ --set webImage={{ ecr }}/linkerd/web \ --set grafana.image.name={{ ecr }}/linkerd/grafana \ --wait \ --timeout 10m0s
After this deployment the linkerd-config ConfigMap get's updated with the right image names, but this is not so efficient, it makes deployment a bit longer than it should be.

So the problem in short is that values file "deepCopied" to linkerd-config ConfigMap, but that ConfigMap's image values for proxy and proxy-init are not the one which was really applied - because Helm post-renderer hooks are not respected.

Can somebody give me some hints how to overcome better this issue?
Thank you.

[Pothulapati]

Hey @gaborho , Thanks for raising the issue.

From what I understand, The problem is that though the image values are changed through post rendering hooks via kustomize, It does not update the configMap which is used with injection on applications, making applications still injected with the older image hosts and not the overriden ones.

Instead of doing a helm upgrade with the settings later... Why can't those settings ( i.e with image overrides) be passed during the initial helm render (which is then updated with helm render webhooks). I don't even think helm post rendering hooks would be necessary if the render command is itself updated to have those image override settings, as they are used everywhere i.e for control-plane and applications.

[gaborho]

Hi @Pothulapati Thank you for your reply. The reason why I could not pass image overrides during the initial helm render, because of the Ansible role I use to identify what images must be re-tagged to ECR. If I set there already the right ECR image names, then I can't do step 3, 4, 5 what I described on my initial comment. Now as you suggested, I introduced one --set overrides for the template render I use to determine what images we need to re-tag, and another set of overrides for the "helm install" or "helm upgrade". This is solving the issue, thanks.

However, I still have one feedback, what would be nice to consider. For almost all components I can use image.name, so I don't need to take care for the tag, I can leave that for the default. However, for prometheus.image, there is no separate .name and .tag field, just image.
So for now, I had to hardcode the default tag:
extra_flags: >- --set-file global.identityTrustAnchorsPEM="{{ certs_dir }}/ca.crt" --set-file identity.issuer.tls.crtPEM="{{ certs_dir }}/issuer.crt" --set-file identity.issuer.tls.keyPEM="{{ certs_dir }}/issuer.key" --set identity.issuer.crtExpiry="{{ issuerCrtExpiryDate.stdout }}" --wait --timeout 10m0s extra_image_flags: >- --set global.proxy.image.name="{{ ecr }}/linkerd/proxy" --set global.proxyInit.image.name="{{ ecr }}/linkerd/proxy-init" --set controllerImage="{{ ecr }}/linkerd/controller" --set webImage="{{ ecr }}/linkerd/web" --set grafana.image.name="{{ ecr }}/linkerd/grafana" --set prometheus.image="{{ ecr }}/prom/prometheus:v2.19.3"

Would be great if this would be also consistent like others, so I can override only the image.name

Thanks for your support.

[Pothulapati]

@gaborho Sorry for the late reply. We actually have updated all the components to follow the same pattern i.e image.name, image.name and image.registry and we also added default values that you can use to override values across {except for prom because its a different registry, but you can still update the field induvidually}. This is currently on the latest edges and will be part of 2.10

https://github.com/linkerd/linkerd2/blob/main/viz/charts/linkerd-viz/values.yaml