Updating Airflow Helm with ArgoCD

[juansgr]

Official Helm Chart version

1.15.0 (latest released)

Apache Airflow version

2.7.1

Kubernetes Version

v1.25.12+26bab08

Helm Chart configuration

values.yaml:

-Home: &-home /opt/- -:
  conn_-: '{{ .Values.data.metadataConnection.protocol }}://{{ .Values.data.metadataConnection.user
    }}:{{ .Values.data.metadataConnection.pass }}@{{ .Values.data.metadataConnection.host 
    }}:{{ .Values.data.metadataConnection.port }}/-_db'
  conn_xcom: '{{ .Values.data.metadataConnection.protocol }}://{{ .Values.data.metadataConnection.user
    }}:{{ .Values.data.metadataConnection.pass }}@{{ .Values.data.metadataConnection.host
    }}:{{ .Values.data.metadataConnection.port }}/{{ .Values.data.metadataConnection.db
    }}'
  
developSourcesVolume:
  enabled: false

jobs:
    enabled: &jobs_enabled true

create-DBJob:
  enabled: true
    ttlSecondsAfterFinished: ~
    command: ~
    args: ["bash", "-c", "exec ./scripts/-.sh"]
    annotations: {}
    jobAnnotations: {}
  securityContext: {}
    securityContexts:
    pod: {}
    container: {}
  containerLifecycleHooks: {}
  serviceAccount:
    automountServiceAccountToken: true
    create: true
    name: ~
        annotations: {}
  resources: {}
    extraContainers: []
  extraVolumeMounts:
    ---
        defaultMode: 0777
  nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
      useHelmHooks: false
  applyCustomEnv: false

createImporterPoolsJob:
  enabled: true
    ttlSecondsAfterFinished: ~
    command: ~

    args:
    - bash
    - -c
    - exec airflow pools import /opt/-/config/importer_pools.json

    annotations: {}
    jobAnnotations: {}
    securityContext: {}
        securityContexts:
    pod: {}
    container: {}
    containerLifecycleHooks: {}
    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~
        annotations: {}
  resources: {}
    extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []
  nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
      useHelmHooks: false
  applyCustomEnv: false

createContentPoolsJob:
  enabled: true
    ttlSecondsAfterFinished: ~
    command: ~

    args:
    - bash
    - -c
    - exec airflow pools import /opt/-/config/content_importer_pools.json

    annotations: {}
    jobAnnotations: {}
    securityContext: {}
        securityContexts:
    pod: {}
    container: {}
    containerLifecycleHooks: {}
    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~
        annotations: {}
  resources: {}
    extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []
  nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
      useHelmHooks: false
  applyCustomEnv: false

fullnameOverride: ""

nameOverride: ""

useStandardNaming: true

revisionHistoryLimit: ~

uid: 10000
gid: 0

securityContext: {}
securityContexts:
  pod: {}
  containers: {}

containerLifecycleHooks: {}

airflowHome: /opt/airflow

defaultAirflowRepository: <repo>

defaultAirflowTag: "1.0"

defaultAirflowDigest: ~

airflowVersion: "2.8.3"

images:
  airflow:
    repository: <repo>
    tag: ~
        digest: ~
    pullPolicy: IfNotPresent
  useDefaultImageForMigration: false
    migrationsWaitTimeout: 60
  pod_template:
    repository: <repo>
    tag: "1.0"
    pullPolicy: IfNotPresent
  flower:
    repository: ~
    tag: ~
    pullPolicy: IfNotPresent
  statsd:
    repository: quay.io/prometheus/statsd-exporter
    tag: v0.22.8
    pullPolicy: IfNotPresent
  redis:
    repository: dockerhub.hi.inet/library/redis
    tag: 7-bullseye
    pullPolicy: IfNotPresent
  pgbouncer:
    repository: dockerhub.hi.inet/apache/airflow
    tag: airflow-pgbouncer-2023.02.24-1.16.1
    pullPolicy: IfNotPresent
  pgbouncerExporter:
    repository: dockerhub.hi.inet/apache/airflow
    tag: airflow-pgbouncer-exporter-2023.02.21-0.14.0
    pullPolicy: IfNotPresent
  gitSync:
    repository: registry.k8s.io/git-sync/git-sync
    tag: v3.6.9
    pullPolicy: IfNotPresent

nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
schedulerName: ~

labels: {}
ingress:
    enabled: ~
  web:
        enabled: true

        annotations: {}

        path: "/"

        pathType: "ImplementationSpecific"

        host: ""

        hosts: ["host"]

        ingressClassName: ""

        tls:
            enabled: false
            secretName: ""

        precedingPaths: []

        succeedingPaths: []

    flower:
        enabled: true

        annotations: {}

        path: "/"

        pathType: "ImplementationSpecific"

        host: ""

        hosts: ["host"]
        ingressClassName: ""

        tls:
            enabled: false
            secretName: ""

networkPolicies:
    enabled: false

airflowPodAnnotations: {}

airflowConfigAnnotations: {}

airflowLocalSettings: |-
  {{- if semverCompare ">=2.2.0" .Values.airflowVersion }}
  {{- if not (or .Values.webserverSecretKey .Values.webserverSecretKeySecretName) }}
  from airflow.www.utils import UIAlert

  DASHBOARD_UIALERTS = [
    UIAlert(
      'Usage of a dynamic webserver secret key detected. We recommend a static webserver secret key instead.'
      ' See the <a href='
      '"https://airflow.apache.org/docs/helm-chart/stable/production-guide.html      'Helm Chart Production Guide</a> for more details.',
      category="warning",
      roles=["Admin"],
      html=True,
    )
  ]
  {{- end }}
  {{- end }}

rbac:
    create: true
  createSCCRoleBinding: true

executor: "CeleryExecutor"

allowPodLaunching: false
  

env: 
  - name: "AIRFLOW__API__AUTH_BACKEND"
    value: "airflow.api.auth.backend.basic_auth"
  - name: "AIRFLOW__LOGGING__LOGGING_LEVEL"
    value: "DEBUG"
  - name: "AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS"
    value: "utils.local_logging.log_config.NEW_LOGGING_CONFIG"
  - name:  "ENV_VAR_LOCAL_LOGGING_LEVEL"
    value: "DEBUG"
  - name: "ENV_VAR_TASK_LOG_LEVEL"
    value: "WARNING"
  - name : "ENV_VAR_COMPONENT_NAME"
    value: "top.enablers.-.airflow"
  - name : "AIRFLOW__LOGGING__CELERY_LOGGING_LEVEL"
    value: "ERROR"
  - name : "AIRFLOW__CORE__PARALLELISM"
    value: "0"
  - name : "AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG"
    value: "32"
  - name : "AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG"
    value: "60"


volumes: []

volumeMounts: []

secret: []
enableBuiltInSecretEnvVars:
  AIRFLOW__CORE__FERNET_KEY: false
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: true
  AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: true
  AIRFLOW_CONN_AIRFLOW_DB: true
  AIRFLOW__WEBSERVER__SECRET_KEY: false
  AIRFLOW__CELERY__CELERY_RESULT_BACKEND: true
  AIRFLOW__CELERY__RESULT_BACKEND: true
  AIRFLOW__CELERY__BROKER_URL: true
  AIRFLOW__ELASTICSEARCH__HOST: true
  AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST: true

priorityClasses: []
extraSecrets: {}
extraConfigMaps: {}
extraEnv: ~
extraEnvFrom: ~
data:
  metadataSecretName: ~
  resultBackendSecretName: ~
  brokerUrlSecretName: ~

    metadataConnection:
    <data>
    resultBackendConnection: ~
  brokerUrl: ~

fernetKey: airflow-fernet-secret
fernetKeySecretName: ~

webserverSecretKey: airflow-webserver-secret
webserverSecretKeySecretName: ~
kerberos:
  enabled: false
  ccacheMountPath: /var/kerberos-ccache
  ccacheFileName: cache
  configPath: /etc/krb5.conf
  keytabBase64Content: ~
  keytabPath: /etc/airflow.keytab
  principal: [email protected]
  reinitFrequency: 3600
  config: |
                            [logging]
    default = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_libs.log"
    kdc = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_kdc.log"
    admin_server = "FILE:{{ template "airflow_logs_no_quote" . }}/kadmind.log"

    [libdefaults]
    default_realm = FOO.COM
    ticket_lifetime = 10h
    renew_lifetime = 7d
    forwardable = true

    [realms]
    FOO.COM = {
      kdc = kdc-server.foo.com
      admin_server = admin_server.foo.com
    }

workers:
    replicas: 4
    revisionHistoryLimit: ~

    command: ~
    args:
    - "bash"
    - "-c"
        - |-
      exec \
      airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery worker" "worker" }}

      livenessProbe:
    enabled: false
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
    command: ~

    updateStrategy: ~
    strategy:
    rollingUpdate:
      maxSurge: "100%"
      maxUnavailable: "50%"

    securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    keda:
    enabled: false
    namespaceLabels: {}

        pollingInterval: 5

            cooldownPeriod: 30

        minReplicaCount: 0

        maxReplicaCount: 10

        advanced: {}
    query: >-
      SELECT ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }})
      FROM task_instance
      WHERE (state='running' OR state='queued')
      {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
      AND queue != '{{ .Values.config.celery_kubernetes_executor.kubernetes_queue }}'
      {{- end }}

            usePgbouncer: false

  persistence:
        enabled: true
        size: 100Gi
        storageClassName:
                fixPermissions: false
        annotations: {}
        securityContexts:
      container: {}
        containerLifecycleHooks: {}

  kerberosSidecar:
        enabled: false
    resources: {}
    securityContexts:
      container: {}
        containerLifecycleHooks: {}

  resources:
    limits:
     cpu: 1500m
     memory: 16Gi
    requests:
     cpu: 500m
     memory: 4Gi

    terminationGracePeriodSeconds: 600

      safeToEvict: true
  extraContainers: []
    extraInitContainers: []

  extraVolumeMounts:
  ---
  extraVolumes:
  ---

    nodeSelector:
    role: apps
  runtimeClassName: ~
  priorityClassName: ~
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
        hostAliases: []

    annotations: {}

  podAnnotations: {}

    labels: {}

  logGroomerSidecar:
        enabled: true
        command: ~
        args: ["bash", "/clean-logs"]
        retentionDays: 15
    resources: {}
    securityContexts:
      container: {}

  waitForMigrations:
        enabled: true
    env: []
        securityContexts:
      container: {}

  env: 
    - name: "AIRFLOW__CELERY__WORKER_CONCURRENCY"
      value: "16"

scheduler:
    hostAliases: []
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
    command: ~

      startupProbe:
    failureThreshold: 6
    periodSeconds: 10
    timeoutSeconds: 20
    command: ~
      replicas: 1
    revisionHistoryLimit: ~

    command: ~
    args: ["bash", "-c", "exec airflow scheduler"]
  updateStrategy: ~
  strategy: ~

      securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    podDisruptionBudget:
    enabled: false

        config:
            maxUnavailable: 1
        resources:
    limits:
     cpu: 600m
     memory: 8Gi
    requests:
     cpu: 300m
     memory: 4Gi

      safeToEvict: true

    extraContainers: []
    extraInitContainers: []
  extraVolumeMounts:
    ---
  extraVolumes:
    ---

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

    annotations: {}

  podAnnotations: {}

    labels: {}

  logGroomerSidecar:
        enabled: true
        command: ~
        args: ["bash", "/clean-logs"]
        retentionDays: 15
    resources: {}
                                securityContexts:
      container: {}
        containerLifecycleHooks: {}

  waitForMigrations:
        enabled: *jobs_enabled
    env: []
        securityContexts:
      container: {}

  env: []

createUserJob:
    ttlSecondsAfterFinished: ~
    command: ~
    args:
    - "bash"
    - "-c"
        - |-
      exec \
      airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "users create" "create_user" }} "$@"
    - --
    - "-r"
    - "{{ .Values.webserver.defaultUser.role }}"
    - "-u"
    - "{{ .Values.webserver.defaultUser.username }}"
    - "-e"
    - "{{ .Values.webserver.defaultUser.email }}"
    - "-f"
    - "{{ .Values.webserver.defaultUser.firstName }}"
    - "-l"
    - "{{ .Values.webserver.defaultUser.lastName }}"
    - "-p"
    - "{{ .Values.webserver.defaultUser.password }}"

    annotations: {}
    jobAnnotations: {}

    labels: {}

    securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    extraContainers: []

  extraVolumes: []
  extraVolumeMounts: []

  nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
      useHelmHooks: false
  applyCustomEnv: false

  env: []

  resources:
    limits:
     cpu: 300m
     memory: 8Gi
    requests:
     cpu: 100m
     memory: 4Gi

migrateDatabaseJob:
  enabled: *jobs_enabled
    ttlSecondsAfterFinished: ~
    command: ~
    args:
    - "bash"
    - "-c"
    - >-
      exec \

      airflow {{ semverCompare ">=2.7.0" .Values.airflowVersion
      | ternary "db migrate" (semverCompare ">=2.0.0" .Values.airflowVersion
      | ternary "db upgrade" "upgradedb") }}

    annotations: {}
    jobAnnotations:
        "argocd.argoproj.io/hook": Sync

    securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

  resources:
    limits:
     cpu: 300m
     memory: 8Gi
    requests:
     cpu: 100m
     memory: 4Gi

    extraContainers: []

  extraVolumeMounts:
    ---
  extraVolumes:
    ---

  nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
      useHelmHooks: false
  applyCustomEnv: false

webserver:
    configMapAnnotations: {}
    hostAliases: []
              allowPodLogReading: true
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
    failureThreshold: 5
    periodSeconds: 10
    scheme: HTTP

  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
    failureThreshold: 5
    periodSeconds: 10
    scheme: HTTP

  startupProbe:
    timeoutSeconds: 20
    failureThreshold: 6
    periodSeconds: 10
    scheme: HTTP

    replicas: 1
    revisionHistoryLimit: ~

    command: ~
    args: ["bash", "-c", "exec airflow webserver"]

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    podDisruptionBudget:
    enabled: false

        config:
            maxUnavailable: 1
          strategy: ~

      securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    extraNetworkPolicies: []
  networkPolicy:
    ingress:
            from: []
            ports:
        - port: "{{ .Values.ports.airflowUI }}"

  resources:
    limits:
     cpu: 1000m
     memory: 8Gi
    requests:
     cpu: 300m
     memory: 4Gi

    defaultUser:
    enabled: *jobs_enabled
    role: Admin
    username: admin
    email: [email protected]
    firstName: admin
    lastName: user
    password: admin

    extraContainers: []
    extraInitContainers: []-
  extraVolumes: []
  extraVolumeMounts: []
  webserverConfig: ~
              webserverConfigConfigMapName: ~

  service:
    type: ClusterIP
        annotations: {}
    ports:
      - name: airflow-ui
        port: "{{ .Values.ports.airflowUI }}"
    loadBalancerIP: ~
                loadBalancerSourceRanges: []

    nodeSelector:
    role: apps
  priorityClassName: ~
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

    annotations: {}

  podAnnotations: {}

    labels: {}

  waitForMigrations:
        enabled: *jobs_enabled
    env: []
        securityContexts:
      container: {}

  env: []

triggerer:
  enabled: true
    replicas: 1
    revisionHistoryLimit: ~

    command: ~
    args: ["bash", "-c", "exec airflow triggerer"]

    updateStrategy: ~
    strategy:
    rollingUpdate:
      maxSurge: "100%"
      maxUnavailable: "50%"

      livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
    command: ~

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

  persistence:
        enabled: true
        size: 100Gi
        storageClassName:
                fixPermissions: false
        annotations: {}

  resources:
    limits:
     cpu: 300m
     memory: 8Gi
    requests:
     cpu: 100m
     memory: 4Gi

    terminationGracePeriodSeconds: 60

      safeToEvict: true

    extraContainers: []
    extraInitContainers: []
  extraVolumes: []
  extraVolumeMounts: []

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

    annotations: {}

  podAnnotations: {}

    labels: {}

  logGroomerSidecar:
        enabled: true
        command: ~
        args: ["bash", "/clean-logs"]
        retentionDays: 15
    resources: {}
                                securityContexts:
      container: {}

        containerLifecycleHooks: {}

  waitForMigrations:
        enabled: *jobs_enabled
    env: []
        securityContexts:
      container: {}

  env: []

    keda:
    enabled: false
    namespaceLabels: {}

        pollingInterval: 5

            cooldownPeriod: 30

        minReplicaCount: 0

        maxReplicaCount: 10

        advanced: {}
    query: >-
      SELECT ceil(COUNT(*)::decimal / {{ .Values.config.triggerer.default_capacity }})
      FROM trigger

dagProcessor:
  enabled: false
    replicas: 1
    revisionHistoryLimit: ~

    command: ~
    args: ["bash", "-c", "exec airflow dag-processor"]

    strategy:
    rollingUpdate:
      maxSurge: "100%"
      maxUnavailable: "50%"

      livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
    command: ~

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

  resources:
    limits:
     cpu: 300m
     memory: 8Gi
    requests:
     cpu: 100m
     memory: 4Gi

    terminationGracePeriodSeconds: 60

      safeToEvict: true

    extraContainers: []
    extraInitContainers: []

  extraVolumes: []
  extraVolumeMounts: []

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

    annotations: {}

  podAnnotations: {}

  logGroomerSidecar:
        enabled: true
        command: ~
        args: ["bash", "/clean-logs"]
        retentionDays: 15
    resources: {}
                          waitForMigrations:
        enabled: true
    env: []

  env: []

flower:
      enabled: true
    revisionHistoryLimit: ~

    command: ~
    args:
    - "bash"
    - "-c"
        - |-
      exec \
      airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery flower" "flower" }}

    extraNetworkPolicies: []
  networkPolicy:
    ingress:
            from: []
            ports:
        - port: "{{ .Values.ports.flowerUI }}"

  resources: {}
                securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    secretName: ~

    username: ~
  password: ~

  service:
    type: ClusterIP
        annotations: {}
    ports:
      - name: flower-ui
        port: "{{ .Values.ports.flowerUI }}"
                        loadBalancerIP: ~
                loadBalancerSourceRanges: []

    extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

    annotations: {}

  podAnnotations: {}

    labels: {}
  env: []

statsd:
    configMapAnnotations: {}

  enabled: true
    revisionHistoryLimit: ~

    args: ["--statsd.mapping-config=/etc/statsd-exporter/mappings.yml"]

    annotations: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

  uid: 65534
      securityContext: {}
          securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

    extraNetworkPolicies: []
  resources: {}
              service:
    extraAnnotations: {}

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

        extraMappings: []

        overrideMappings: []

  podAnnotations: {}
  env: []

pgbouncer:
    enabled: false
    replicas: 1
    revisionHistoryLimit: ~
    command: ["pgbouncer", "-u", "nobody", "/etc/pgbouncer/pgbouncer.ini"]
    args: ~
  auth_type: md5
  auth_file: /etc/pgbouncer/users.txt

    annotations: {}

  podAnnotations: {}

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    extraNetworkPolicies: []

    metadataPoolSize: 10
  resultBackendPoolSize: 5

    maxClientConn: 100

  configSecretName: ~

    podDisruptionBudget:
    enabled: false

        config:
            maxUnavailable: 1
        resources: {}

  service:
    extraAnnotations: {}

    verbose: 2
  logDisconnections: 1
  logConnections: 1

  sslmode: "prefer"
  ciphers: "normal"

  ssl:
    ca: ~
    cert: ~
    key: ~

      extraIniMetadata: ~
  extraIniResultBackend: ~
    extraIni: ~

  extraVolumes: []
  extraVolumeMounts: []

    extraContainers: []

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

  uid: 65534

    securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks:
    preStop:
      exec:
                command: ["/bin/sh", "-c", "killall -INT pgbouncer && sleep 120"]

  metricsExporterSidecar:
    resources: {}
                            sslmode: "disable"

        statsSecretName: ~

        statsSecretKey: ~

        securityContexts:
      container: {}

        containerLifecycleHooks: {}

    livenessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1

    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1

    env: []

redis:
  enabled: true
  terminationGracePeriodSeconds: 600

    serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

  persistence:
        enabled: true
        size: 1Gi
        storageClassName:
        annotations: {}

  resources:
    limits:
     cpu: 300m
     memory: 8Gi
    requests:
     cpu: 100m
     memory: 4Gi

    passwordSecretName: ~

        password: ~

      safeToEvict: true

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

    uid: 0
    securityContext: {}
        securityContexts:
    pod:
      fsGroup: 0
      runAsGroup: 0
      runAsUser: 999
    container:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL

    containerLifecycleHooks: {}

  podAnnotations: {}
registry:
  secretName: crvpprevrm

            elasticsearch:
    enabled: false
    secretName: ~
                connection: {}

ports:
  flowerUI: 5555
  airflowUI: 8080
  workerLogs: 8793
  triggererLogs: 8794
  redisDB: 6379
  statsdIngest: 9125
  statsdScrape: 9102
  pgbouncer: 6543
  pgbouncerScrape: 9127

quotas: {}

limits: []

cleanup:
  enabled: false
    schedule: "*/15 * * * *"
  command: ~
    args: ["bash", "-c", "exec airflow kubernetes cleanup-pods --namespace={{ .Release.Namespace }}"]

    jobAnnotations: {}

    nodeSelector:
    role: apps
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  podAnnotations: {}

    labels: {}

  resources: {}
                serviceAccount:
            automountServiceAccountToken: true
        create: true
            name: ~

        annotations: {}

    securityContext: {}
      env: []

    securityContexts:
    pod: {}
    container: {}

    containerLifecycleHooks: {}

      failedJobsHistoryLimit: ~
  successfulJobsHistoryLimit: ~

postgresql:
  enabled: false
  image:
    tag: "11"
  auth:
    enablePostgresUser: true
    postgresPassword: postgres
    username: ""
    password: ""

config:
  core:
    dags_folder: '/opt/-/dags'
    dags_are_paused_at_creation: 'False'
        load_examples: 'False'
    executor: '{{ .Values.executor }}'
        colored_console_log: 'False'
    remote_logging: '{{- ternary "True" "False" .Values.elasticsearch.enabled }}'
  logging:
    remote_logging: '{{- ternary "True" "False" .Values.elasticsearch.enabled }}'
    colored_console_log: 'False'
  metrics:
    statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
    statsd_port: 9125
    statsd_prefix: airflow
    statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
  webserver:
    enable_proxy_fix: 'True'
        rbac: 'True'
  celery:
    flower_url_prefix: '{{ ternary "" .Values.ingress.flower.path (eq .Values.ingress.flower.path "/") }}'
    worker_concurrency: 16
  scheduler:
    standalone_dag_processor: '{{ ternary "True" "False" .Values.dagProcessor.enabled }}'
        statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
    statsd_port: 9125
    statsd_prefix: airflow
    statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
        run_duration: 41460
  elasticsearch:
    json_format: 'True'
    log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
  elasticsearch_configs:
    max_retries: 3
    timeout: 30
    retry_timeout: 'True'
  kerberos:
    keytab: '{{ .Values.kerberos.keytabPath }}'
    reinit_frequency: '{{ .Values.kerberos.reinitFrequency }}'
    principal: '{{ .Values.kerberos.principal }}'
    ccache: '{{ .Values.kerberos.ccacheMountPath }}/{{ .Values.kerberos.ccacheFileName }}'
  celery_kubernetes_executor:
    kubernetes_queue: 'kubernetes'
      kubernetes:
    namespace: '{{ .Release.Namespace }}'
        airflow_configmap: '{{ include "airflow_config" . }}'
    airflow_local_settings_configmap: '{{ include "airflow_config" . }}'
    pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
    worker_container_repository: <repo>
        multi_namespace_mode: '{{ ternary "True" "False" .Values.multiNamespaceMode }}'
    kubernetes_executor:
    namespace: '{{ .Release.Namespace }}'
    pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
        worker_container_repository: <repo>
    multi_namespace_mode: '{{ ternary "True" "False" .Values.multiNamespaceMode }}'
  triggerer:
    default_capacity: 1000
multiNamespaceMode: false
podTemplate: ~
dags:
  persistence:
        annotations: {}
        enabled: false
        size: 1Gi
        storageClassName:
        accessMode: ReadWriteOnce
        existingClaim:
        subPath: ~
  gitSync:
    enabled: false

                repo: https://github.com/apache/airflow.git
    branch: v2-2-stable
    rev: HEAD
    depth: 1
        maxFailures: 0
            subPath: "tests/dags"
    wait: 5
    containerName: git-sync
    uid: 65533

        securityContext: {}
            securityContexts:
      container: {}

        containerLifecycleHooks: {}

                        extraVolumeMounts: []
    env: []
                resources: {}
                        logs:
  useDevelopPVC: false
  persistence:
        enabled: true
        size: 100Gi
        annotations: {}
        storageClassName: "-"
        existingClaim: airflow-logs

Docker Image customizations

No response

What happened

I'm deploying Apache Airflow Helm Chart on our corporate environment with ArgoCD and I would love to get some help on something that I'm not being able to solve. Thanks in advance.

So far it's working pretty good, I've made some customizations to the deployment, pv's, added some jobs, etc. The problem is, when upgrading with ArgoCD, some barriers show up.

Firstly, fernet-key didn't allow upgrade on objects claiming error "missing fernet-key", so I disabled that temporarily for debug purposes. Now, everytime we commit into the branch that ArgoCD is deploying, the deploy breaks, being unable to fully sync the app, claiming errors of "Inmutable fields". I think the app is not trying to create a new updated object when we try to update, and instead is trying to modify the already existing objects. The troubleshoting I've found is just manually deleting the app and letting ArgoCD deploy it entirely again, but this way we are loosing ArgoCD automatization point.

I’m sure I’m not the only one deploying this Helm chart with ArgoCD, so I’d like to know if anyone can spot any visible errors or missing configurations. So far, I’ve followed the documentation at https://airflow.apache.org/docs/helm-chart/1.8.0/ to deploy with ArgoCD, but I might be overlooking something.

What you think should happen instead

ArgoCD should be able to update the deployment configuration and images when we push into the deployed branch.

How to reproduce

In an OpenShift cluster with ArgoCD, we deployed the official Helm chart via ArgoCD, using the configurations previously described in the values.yaml.

The deployment is successful, and the application works correctly. ArgoCD is able to deploy all objects and identify the synchronization status of the app.

We then attempt to make a change to the deployment, either by committing to the deployed branch or switching branches. The change could involve modifying a configuration or updating the image version.

Synchronization via ArgoCD fails. The only solution is to delete the deployment and let ArgoCD recreate the app from scratch.

Anything else

Thanks in advance!

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Converted it into discussion as this is not Airlfow issue - just question how to work with ArgoCD - hopefully indeed someone who works with ArgoCD will help you - you can also look through issues and discussions here with ArgoCD, maybe you will find something.