Unable to add custom attributes under service

[fabry00]

Describe the bug
I'd like to add several attributes under service.* but so far all the attributes that I've added in the processors.resource.atrtibutes are added as children of labels.*

Steps to reproduce

Given the following configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  resource:
    attributes:
    - key: deployment.environment
      action: insert
      value: dev
    - key: service.cluster
      value: XXXX
      action: insert
    - key: cluster
      value: XXXX
      action: insert

exporters:
  debug:
    verbosity: detailed

....
....

The attribute service.cluster will be injected as labels.service_cluster while cluster in labels.cluster

What did you expect to see?
service.cluster: XXXX

What did you see instead?
labels.service_cluster:XXXX or
labels.cluster: XXXX

What version did you use?
0.108.0

Environment
openshift

Additional context

ScopeSpans SchemaURL: 
...
Span #0
    Trace ID       : 2760dc4ca9976b747cc7752ee79f36b9
    Parent ID      : 
    .....
    Status message : 
Attributes:
     -> http.scheme: Str(https)
     -> http.client_ip: Str(172.16.8.1)
     -> kong.request.id: Str(f71b580eba670e4c66d6902551bcdd61)
     -> http.method: Str(GET)
     .....   
ResourceSpans #1
Resource SchemaURL: 
Resource attributes:
     -> service.name:xxxxx
     -> service.version: xxxxx
     -> service.instance.id: xxxxx
     -> deployment.environment: Str(dev)
     -> service.cluster: xxxx
     -> customer: xxxxx

[dmathieu]

I'm not sure I get what you mean here. In your debugexporter output, the resources is properly specified as service.cluster.
Also, OTel has no such thing as "labels".

Could this be the way the visualization tool you're using handles attributes that aren't semantic conventions?

[fabry00]

Hi @dmathieu ,
thank you for looking into this. I am forwarding the traces to elasticsearch, and this is what I see in elasticsearch

{
  "@timestamp": [
    "2024-11-22T09:10:27.608Z"
  ],
  "agent.name": [
    "otlp"
  ],
  "agent.name.text": [
    "otlp"
  ],
  "agent.version": [
    "unknown"
  ],
  "data_stream.dataset": [
    "apm"
  ],
  "data_stream.namespace": [
    "dev"
  ],
  "data_stream.type": [
    "traces"
  ],
  "event.outcome": [
    "success"
  ],
  "event.success_count": [
    1
  ],
  "labels.service_cluster": [
    "xxxx"
  ],
  "labels.service_customer": [
    "yyyy"
  ],
  "labels.service_ldc": [
    "eu"
  ],
  "labels.service_listner": [
    "internal"
  ],
  "observer.hostname": [
    "my-fleetserver-5d4cd5765f-b4gpb"
  ],
  "observer.type": [
    "apm-server"
  ],
  "observer.version": [
    "8.16.0"
  ],
  "parent.id": [
    "174e90f23b7d7ee7"
  ],
  "processor.event": [
    "span"
  ],
  "service.environment": [
    "dev"
  ],
  "service.framework.name": [
    "my-internal"
  ],
  "service.framework.name.text": [
    "my-internal"
  ],
  "service.framework.version": [
    "0.1.0"
  ],
  "service.language.name": [
    "unknown"
  ],
  "service.language.name.text": [
    "unknown"
  ],
  "service.name": [
    "aaaaa_my_my-mytool_v1"
  ],
  "service.name.text": [
    "aaaaa_my_my-mytool_v1"
  ],
  "service.node.name": [
    "416cab84-c2b0-492c-92db-545bf4b760a4"
  ],
  "service.node.name.text": [
    "416cab84-c2b0-492c-92db-545bf4b760a4"
  ],
  "service.version": [
    "3.8.1.0"
  ],
  "span.duration.us": [
    1
  ],
  "span.id": [
    "55c6f47d03a7274a"
  ],
  "span.name": [
    "my.header_filter.plugin.post-function"
  ],
  "span.name.text": [
    "my.header_filter.plugin.post-function"
  ],
  "span.representative_count": [
    1
  ],
  "span.subtype": [
    "internal"
  ],
  "span.type": [
    "app"
  ],
  "tags": [
    "_geoip_database_unavailable_GeoLite2-City.mmdb"
  ],
  "timestamp.us": [
    1732266627608141
  ],
  "trace.id": [
    "e9cb83601e16e99346433d27a11b417a"
  ],
  "_id": "pngjU5MBh-RdWrIqEuv7",
  "_index": ".ds-traces-apm-dev-2024.11.21-000031",
  "_score": null
}

So you think that it is elasticsearch? I haven't thought about that

[dmathieu]

Disclaimer: I work for elastic.

Yes, this is the behavior of the OTLP endpoints in apm-server and our service offering.
You may want to switch to the elasticsearch exporter instead, which doesn't append names to resources and attributes.
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter
You may also want to look at that guide.