[operator] service "opentelemetry-operator-webhook" not found

[davgia]

Hi everyone, I have disabled the webhook as suggested by the documentation. The helm chart installation completes successfully without errors but I do not see pods created after I apply the collector definition. Moreover when I try to list opentelemetrycollector resources from kubectl I receive the following error:

kubectl get opentelemetrycollector --all-namespaces
Warning: OpenTelemetryCollector v1alpha1 is deprecated. Migrate to v1beta1.
Error from server: conversion webhook for opentelemetry.io/v1beta1, Kind=OpenTelemetryCollector failed: Post "https://opentelemetry-operator-webhook.monitoring.svc:443/convert?timeout=30s": service "opentelemetry-operator-webhook" not found

Helm chart:

# helm upgrade --install opentelemetry-operator open-telemetry/opentelemetry-operator --version "0.60.0" -f .\values.yaml -n monitoring
manager:
  collectorImage:
    repository: "otel/opentelemetry-collector-k8s"
  # disable webhooks
  env:
    ENABLE_WEBHOOKS: "false"
  # resources configuration
  resources:
    limits:
      memory: 64Mi
    requests:
      cpu: 5m
      memory: 32Mi
# disable admission webhooks
admissionWebhooks:
  create: false

Collector definition:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: "${kubernetes_namespace_v1.environment.id}"
spec:
  mode: deployment
  config:
    receivers:
      jaeger:
        protocols:
          grpc: {}
          thrift_http: {}
          thrift_compact: {}
      otlp:
        protocols:
          grpc: {}
          http: {}
      zipkin: {}
    extensions:
      health_check: {}
      pprof: {}
      zpages: {}
    processors:
      batch: {}
    exporters:
      otlp:
        endpoint: tempo.monitoring:4317
        tls:
          insecure: true
    service:
      extensions:
        - health_check
        - pprof
        - zpages
      pipelines:
        traces:
          receivers:
            - otlp
            - jaeger
            - zipkin
          processors:
            - batch
          exporters:
            - otlp

Can someone help me to pinpoint the error or the misconfiguration?

Thanks in advance

[jaronoff97]

@davgia I think Kubernetes is attempting to convert them using webhooks but because you have disabled webhooks the operator will just fail. I would recommend amending your CRD definition to remove the conversion webhook if its the case that you do not have and never will have v1alpha1 collector resources. You can also just install the CRD directly from here.

[davgia]

@davgia I think Kubernetes is attempting to convert them using webhooks but because you have disabled webhooks the operator will just fail. I would recommend amending your CRD definition to remove the conversion webhook if its the case that you do not have and never will have v1alpha1 collector resources. You can also just install the CRD directly from here.

Thank you @jaronoff97 for the input.

I am in a clean cluster and I've installed the operator so I guess the CRDs are already at the latest version. I don't particularly like changing the CRD definition manually (I'm automating the procedure with Terraform, this CRD change cannot be done with it I think). I may try to enable webhook with the following strategy (straight from operator documentation):

You can use an automatically generated self-signed certificate by setting admissionWebhooks.certManager.enabled to false and admissionWebhooks.autoGenerateCert.enabled to true. Helm will create a self-signed cert and a secret for you.

I will come back to you if this fixes the problem. Last chance is to install cert-manager, but there are scenarios in which I don't have it already installed (and install it just for this seems overkill to me).

[jaronoff97]

Yep, i believe this is a flavor of #677 actually where you just want the simple CRD installation. I.e. i'm working on it 😓

[davgia]

I must clarify that I still need the operator to create the collector (I am also using the auto-instrumentation, but I've removed it from this issue for simplification). Also I will add that I've deployed the operator in other cluster over an old version (0.40.0) that was working fine (configured without webhook) and stopped working (even if the chart is upgraded correctly).

[jaronoff97]

yeah i believe that would be because of the CRD change. I would recommend setting crds.create=false for that cluster and rely on managing the CRDs externally.

[jaronoff97]

would it be okay to close this in favor of #677 and #1201?

[davgia]

Ok, I will close this. Thank you.