Operator controller seems stuck, changes not getting applied

[charlieevett]

I'm seeing a persistent error in the controller manager pod:

2023-03-28T20:23:17.540216878Z 1.6800349975401418e+09 INFO Starting workers {"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "worker count": 1} 2023-03-28T20:23:18.705601874Z 1.680034998705451e+09 ERROR Reconciler error {"controller": "emqxbroker", "controllerGroup": "apps.emqx.io", "controllerKind": "EmqxBroker", "emqxBroker": {"name":"emqx","namespace":"mqtt-test"}, "namespace": "mqtt-test", "name": "emqx", "reconcileID": "c383e54c-fc0c-4ebd-bb2a-bcea6c2a65f6", "error": "failed to calculate patch for Service emqx: Failed to generate merge patch: Failed to generate merge patch: Invalid JSON Document"} 2023-03-28T20:23:18.705624534Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem 2023-03-28T20:23:18.705627284Z /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 2023-03-28T20:23:18.705629114Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 2023-03-28T20:23:18.705630714Z /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

I've been trying to make changes to the EmqxBroker definition to set up TLS using extraVolumes and extraVolumeMounts, but my changes don't seem to lead anything happening. I've fallen back to just trying to add a second emptyDir() just to see if I can make something update, to no avail. Perhaps some earlier issue is causing controller manager to be stuck on previous changes and it can't move on to my current work. ?

Any suggestions for troubleshooting?

[Rory-Z]

Could you please show the YAML file for EMQX custom resource ?

[charlieevett]

apiVersion: apps.emqx.io/v1beta3
kind: EmqxBroker
metadata:
  labels:
    apps.emqx.io/instance: emqx
    apps.emqx.io/managed-by: emqx-operator
  name: emqx
  namespace: mqtt-test
spec:
  emqxTemplate:
    acl:
    - '{allow, all}.'
    args:
    - bash
    - -c
    - |
      printenv | grep "^EMQX_"
      emqx foreground
    config:
      cluster.discovery: dns
      cluster.dns.app: emqx
      cluster.dns.name: emqx-headless.mqtt-test.svc.cluster.local
      cluster.dns.type: srv
      listener.ssl.external: "8883"
      listener.tcp.external: "1883"
      listener.tcp.internal: "11883"
      listener.ws.external: "8083"
      listener.wss.external: "8084"
      log.to: both
      name: emqx
    extraVolumeMounts:
    - mountPath: /tmp/fake
      name: fake-volume
    extraVolumes:
    - emptyDir: {}
      name: fake-volume
    image: emqx/emqx:4.4.8
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /status
        port: 8081
      initialDelaySeconds: 60
      periodSeconds: 30
    modules:
    - enable: true
      name: emqx_mod_acl_internal
    - enable: true
      name: emqx_mod_presence
    readinessProbe:
      failureThreshold: 12
      httpGet:
        path: /status
        port: 8081
      initialDelaySeconds: 10
      periodSeconds: 5
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 125m
        memory: 64Mi
    securityContext:
      fsGroup: 1000
      fsGroupChangePolicy: Always
      runAsGroup: 1000
      runAsUser: 1000
    serviceTemplate:
      metadata:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-attributes: deletion_protection.enabled=true
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:838317581417:certificate/1c18426f-7668-452e-8b7e-428f3918520b
          #service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 1883
          service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
          service.beta.kubernetes.io/aws-load-balancer-type: nlb
        labels:
          apps.emqx.io/instance: emqx
          apps.emqx.io/managed-by: emqx-operator
        name: emqx
        namespace: mqtt-test
      spec:
        ports:
        - name: http-management-8081
          port: 8081
          protocol: TCP
          targetPort: 8081
        - name: mqtt-tcp-1883
          nodePort: 30649
          port: 1883
          protocol: TCP
          targetPort: 1883
        - name: http-dashboard-18083
          nodePort: 30648
          port: 18083
          protocol: TCP
          targetPort: 18083
        - name: mqtt-tcp-11883
          nodePort: 30650
          port: 11883
          protocol: TCP
          targetPort: 11883
        - name: mqtt-ws-8083
          nodePort: 30651
          port: 8083
          protocol: TCP
          targetPort: 8083
        - name: mqtt-wss-8084
          nodePort: 30652
          port: 8084
          protocol: TCP
          targetPort: 8084
        selector:
          apps.emqx.io/instance: emqx
          apps.emqx.io/managed-by: emqx-operator
        type: LoadBalancer

[Rory-Z]

Hi, @charlieevett could you please delete this line and retry ?

          #service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 1883

Maybe "#" cannot be encoded in JSON.

[charlieevett]

Thanks, I tried taking that out and applying it again:

$ kubectl apply -f emqx-broker.yaml

but I get the same error. One thing is that I see that error in the logs every 15 minutes or so, like it just keeps retrying. Which has me wondering if it's some other old version that it is retrying and not the latest thing that I set.

[Rory-Z]

Very strange. I noticed that you are using v1beta3, which is an old APIVersion. Could you update to the latest EMQX Operator and use v1beta4 retry it ? v1beta4 is different from v1beta3, you can check this: https://github.com/emqx/emqx-operator/tree/main/config/samples/emqx/v1beta4

[charlieevett]

I'm not certain but I was under the impression that we were stuck with v1beta3 because our Kubernetes cluster is version 1.20. We have plans to build a new cluster with a current version of k8s but not for a bit.

[Rory-Z]

I understand, can you tell me the specific version of EMQX Operator?

[charlieevett]

Working on trying to find this. How would I figure out which version of the operator we have installed?

[Rory-Z]

You can running this command kubectl -n emqx-operator-system get deployments.apps emqx-operator-controller-manager -o json | jq '.spec.template.spec.containers[0].image'

[charlieevett]

Thanks! I get: "emqx/emqx-operator-controller:1.2.1"

[Rory-Z]

This is a very old version. We will check it. Please wait sometime.

[Rory-Z]

Hi @charlieevett, I'm use your YAML file and emqx/emqx-operator-controller:1.2.1 running in Minikube, just have an error is spec.ports[1].nodePort: Invalid value: 32769: provided port is not in the valid range. The range of valid ports is 30000-32767", when I fix this, everything is good.
Did I miss anything?

[charlieevett]

Thanks for your work on this, I really appreciate it. I don't see 32769 in my YAML file. I wonder were it is coming from?

[Rory-Z]

Sorry for delay, this is a bug in EMQX Operator 1.2.1. When use NodePort service type and doesn't set .spec.ports[].nodePort, EMQX Operator may use an out-of-range port to update the service

[charlieevett]

Thanks. We're going to try upgrading the operator and proceeding from there.

[Rory-Z]

So cool, could you mark as answer for this question ?