Image Updater With Multiple Sources - Successfully updated image but pending spec update (dry run=false)

[TitanRob16]

Describe the bug
We have a repo containing the Helm Chart we want to deploy from GitLab, and another repo containing the values .yaml files for environment (lab, dev, uat, prd).

After setting the argocd-image-updater annotations for the ApplicationSet through Terraform, Image Updater recognises the need to update the spec image, but doesn't do so with little to no information in the logs. All we see is:
Successfully updated image but pending spec update (dry run=false)

This happens even when we set the deployment.image.tag in our values to an empty string in the hopes that Argo Image Updater will recognise the need for 'latest' or the highest 'semver' and update the deployment.

To Reproduce
Steps to reproduce the behavior:

resource "kubernetes_manifest" "dev_application_set" {
  count = local.env == "dev" || local.env == "qa" || local.env == "uat" || local.env == "prd" ? 1 : 0
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "ApplicationSet"
    metadata = {
      name      = "dev-microservices"
      namespace = local.argocd_namespace
      labels = {
        "company.dev/domain" = local.dev.name
        "company.dev/squad"  = local.dev.name
        "companydev/chart"  = "${local.dev.name}-app-of-apps"
      }
    }
    spec = {
      goTemplate        = true
      goTemplateOptions = ["missingkey=error"]
      generators = [
        {
          git = {
            repoURL  = local.dev.values_repoURL
            revision = "main"
            files = [
              { path : "applications/values/${local.env}/*.yaml" },
            ]
          }
        }
      ]
      template = {
        metadata = {
          name = "{{ .global.chartName }}" 
          annotations = {
            "argocd-image-updater.argoproj.io/image-list"                  = "job-service={{ .deployment.image.repository }}"
            "argocd-image-updater.argoproj.io/job-service.update-strategy" = "semver"
            "argocd-image-updater.argoproj.io/argocd.force-update"         = "true"
            "argocd-image-updater.argoproj.io/job-service.helm.image-name" = "deployment.image.repository"
            "argocd-image-updater.argoproj.io/job-service.helm.image-tag"  = "deployment.image.tag"
          }
          labels = {
            "company.dev/domain" = local.dev.name
            "company.dev/squad"  = local.dev.name
            "company.dev/chart"  = "${local.dev.name}-microservices"
          }
        }
        spec = {
          project = local.gke_cluster_deployments[0].project

          sources = [
            {
              repoURL        = local.dev.values_repoURL
              targetRevision = "main"
              ref            = "values"
            },
            {
              chart          = ""
              repoURL        = local.dev.repoURL
              path           = "applications/chart"
              targetRevision = "main"
              helm = {
                ignoreMissingValueFiles = true
                valueFiles = [
                  "$values/applications/values/defaults/${local.env}-values.yaml",
                  "$values/applications/secrets/defaults/${local.env}-secrets.yaml",
                  "$values/applications/values/${local.env}/{{ .path.filename }}",
                  "$values/applications/secrets/${local.env}/{{ .path.filename }}",
                ]
              }
            }
          ]
          destination = {
            name      = local.gke_cluster_deployments[0].destination
            namespace = local.dev.namespace
          }
          syncPolicy = {
            automated = {
              prune    = true
              selfHeal = true
            }
            syncOptions = ["CreateNamespace=true"]
          }
        }
      }
    }
  }
}

Expected behavior
Argo CD Image Updater should see the latest image and update the spec of the deployments, or at least provide better errors and information so we can debug this further.

Version
v2.12.6+4dab5bd

Logs

time="2024-12-02T16:22:04Z" level=debug msg="found 100 from 100 tags eligible for consideration" image=europe-docker.pkg.dev/company-artifact-registry-dev/dev/titan/job-service
time="2024-12-02T16:22:04Z" level=info msg="Setting new image to europe-docker.pkg.dev/company-artifact-registry-dev/dev/titan/job-service:latest" alias=job-service application=job-service image_name=company-artifact-registry-dev/dev/titan/job-service image_tag= registry=europe-docker.pkg.dev
time="2024-12-02T16:22:04Z" level=debug msg="target parameters: image-spec= image-name=deployment.image.repository, image-tag=deployment.image.tag" application=job-service image=europe-docker.pkg.dev/company-artifact-registry-dev/dev/titan/job-service namespace=argocd
time="2024-12-02T16:22:04Z" level=info msg="Successfully updated image 'europe-docker.pkg.dev/company-artifact-registry-dev/dev/titan/job-service' to 'europe-docker.pkg.dev/company-artifact-registry-dev/dev/titan/job-service:latest', but pending spec update (dry run=false)" alias=job-service application=job-service image_name=company-artifact-registry-dev/dev/titan/job-service image_tag= registry=europe-docker.pkg.dev
time="2024-12-02T16:22:04Z" level=debug msg="Using commit message: "
time="2024-12-02T16:22:04Z" level=info msg="Committing 1 parameter update(s) for application job-service" application=job-service

[chengfang]

the similar issue was also reported in #423 (comment) and #941 Will look into it.