Health Collector not working with mTLS

[ssurbhi19]

Describe the bug
All collectors are configured to send their internal metrics to the Health Collector on port 8889. A PodMonitor is used to scrape these metrics and forward them to the Health Collector. However, when mTLS is set to STRICT mode within the Istio service mesh, the Health Collector fails to scrape the metrics. Notably, the data flows as expected when mTLS is in PERMISSIVE mode combined with a network policy, but it encounters issues in STRICT mode.

Steps to reproduce

1. Create a Gateway Collector that sends its internal metrics at port 8889.
2. Create a Pod Montior to scrape the metrics from the specified port.
3. Enable mTLS in STRICT mode in Health and Gateway Collector

What did you expect to see?
Traces/Metrics/Logs should flow from Gateway Collector to Health Collector without any error.

What did you see instead?

Error in Health Collector::
2024-11-19T09:01:16.486Z warn internal/transaction.go:128 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1732006876483, "target_labels": "{name="up", app_kubernetes_io_name="x-collector", container="otc-container", endpoint="metrics", instance="x:8889", job="ns/collector-health", namespace="ns", pod="x-collector-nnz7r", service_component="x-collector", service_name="x-collector-nnz7r"}"}

What version did you use?

• Collector image: "open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib" Tag: "0.111.0"
• Target Allocator image: "open-telemetry/opentelemetry-operator/target-allocator" Tag: "v0.111.0"

What config did you use?

exporters:
  debug:
    verbosity: basic
  prometheusremotewrite/grafana_metrics:
    endpoint: >-
      http://xyz.ns.svc:80/api/v1/push
    remote_write_queue:
      enabled: false
    resource_to_telemetry_conversion:
      enabled: true
    retry_on_failure:
      enabled: false
  signalfx:
    access_token: ${token}
    api_tls:
      insecure_skip_verify: true
    ingest_tls:
      insecure_skip_verify: true
    max_idle_conns: 80
    realm: us1
    timeout: 5s
    tls:
      insecure_skip_verify: true

Environment
MacOS - Sonoma 14.7

** Additional context **