Thanks for the proposal!

I think we really need some way to capture large data that, somewhere in the pipeline, can be uploaded to a storage that's better designed for such data.

A few feedback points on the details:

1. We avoid having an unbound data in attribute values in semantic conventions. We don't have http.request|response.body.content defined and if someone proposed such attributes, I'd recommend defining request/response contents as events. One exception is log.record.original which might in theory be huge.

   I'd also recommend users put any large content into the event bodies - events volume could be controlled using common logging verbosity if someone does not want them, but for attributes someone would need to come up with custom feature-flags or drop attributes in a custom processor wasting performance on collecting them.

   So I think we need a solution for uploading an event body (or a specific jpath in the body) but it could be the same solution as for uploading an attribute value if that's also important.

2. How:

   • I'd entertain the idea of putting ref. first to avoid collision. I.e. gen_ai.prompt would become ref.gen_ai.prompt. We can also document root ref (or similar) namespace for it. Or adding _ref suffix (e.g. gen_ai.prompt_ref) which would also not result in a collision.
   • Since we need this mechanism for events too, we might need to rename the event as a result since we're going to change the event structure. E.g. gen_ai.choice event would become ref.gen_ai.choice. The alternative would be to keep the same event name and just replace the removed value with null and add a reference as an attribute.
   • We'll need to figure out what to do with the original value. Perhaps it could be configurable and done by the processor.

In any case, I'm really supportive of this proposal with a caveat that large data should not be populated on span attributes and we should not encourage people to do it.

Thanks for the support of the general direction lmolkova!

Event body

I think we need a solution for uploading an event body (or a specific jpath in the body)

I emphatically agree. I would like it to be possible to perform this kind of write-aside solution with any relevant signal type (not just attributes, not just in spans). That would include uploading a subfield of the event body (targeted using a JSON Path, JMESPath, or similar). Ideally, this would be accomplished with the data model and implementation.

I'd also recommend users put any large content into the event bodies … a caveat that large data should not be populated on span attributes and we should not encourage people to do it

Our team is fully supportive of following OTel conventions and encouraging others to follow OTel conventions, so I don't think there is any conflict here in terms of what we recommend to our customers. Our team will aim to optimize the cases that follow OTel conventions and will encourage our customers to adopt said conventions, because they align with the "well lit path" in our systems which we prioritize. If the convention says that this data should be placed in an event body rather than in span attributes, then that is what we will encourage and do.

Original value

We'll need to figure out what to do with the original value

I agree with you that it ought to be configurable. Potential options might be to drop it entirely before sending to the telemetry destination or to truncate it (possibly with redaction) before sending along to the telemetry endpoint. The latter would make it easier for a telemetry backend to provide users with a preview of the content before linking to the full uploaded document.

Attribute naming

I think that there are a lot of different valid naming schemes. Some things which I consider important to the naming are:

1. Ability to add metadata - the URI is all that should be mandatory, but I think it would be ideal if the naming scheme were flexible to being able to provide additional metadata about the object that was uploaded like content type, size, a hashing algorithm, hash.

2. Implementation efficiency / flexibility - we should choose a naming structure/scheme that least constrains implementation and enables highly efficient implementation

3. Not colliding with existing practices - we should ensure that whatever naming we choose does not overlap with any existing usage/names.

Between the two options you suggested (ref. as a prefix vs _ref as a suffix), I am more favorable to _ref as a suffix, because I think it might be useful for the reference attribute to sort along with the attribute it replaces. This way, an implementation that uses binary search to find attributes could minimize the amount of searching to locate the nearby _ref attributes.

However, I am somewhat concerned that an attribute ending in _ref could already exist (in vendor-specific or customer-specific attributes not captured in OTel Semantic Conventions documentation). I'm also wondering about the ability to add other metadata about the reference; would we, for example, need to ban the use of _ref_ anywhere in attribute names in order to support a _ref_uri, _ref_content_type, and _ref_metadata (for any new metadata) suffixes?

Event Naming

I was assuming that event names would remain unchanged and that event handlers should be able to gracefully handle missing expected attributes (and optionally add support for handling reference attributes in place of the original attributes). But perhaps I am missing something. Can you share more regarding why the event name would need to be changed? Is there a particular breakage to existing backends that you would expect this to cause? (Shouldn't backends already gracefully handle the case where expected attributes are absent?)

fwiw, while we do have some legacy span attributes that may contain large values, we have since clarified that the intention of attributes is for indexing, not for storing large blobs of data. It should be possible to store attributes in a column store without concern that this will create problems with values measured in megabytes.

Any solution that can avoid putting large values in attributes should be avoided, and we should focus on making these values event payloads.

Any solution that can avoid putting large values in attributes should be avoided

Agreed. This will help us deal with "legacy span attributes that may contain large values", by making them into smaller values (just a URI and possibly other metadata like a content-type) so that backends don't need to deal with the possibility of very large attribute values. [But, indeed, we should encourage a different approach for new data].

we should focus on making these values event payloads

Agreed. However, it is worth noting that this doesn't magically solve all of the problems for all backends. Some backends may also have limits on the size of such payloads, too. [Especially where GenAI use cases begin introducing things like entire PDFs, images, etc. in prompt/response logging].

Since event payloads and attributes use a common data model (AnyValue), I'm thinking that it should ideally be possible to use a common data model to describe a write-aside solution that uploads targeted pieces of the attributes and/or event bodies so that the particular signal type or field path in the signal isn't pertinent to the solution.

Just checking back in ... what are the next steps here?

It looks like the next step is to actually propose the update.

There is a pending PR in #1521

It was brought up that VCS already uses .ref., so we will need a different segment.

Let's discuss the specific name alternatives in the pull request in #1521