ddtrace/opentelemetry: add RWMutex to handle concurrent calls to setters

[darccio]

What does this PR do?

Fixes #2518.

Motivation

Making sure OTel spans can be used concurrently, as our tracer.Span.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

For Datadog employees:

• If this PR touches code that handles credentials of any kind, such as Datadog API keys, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-26 19:24:26

Comparing candidate commit d4bb929 in PR branch dario.castane/2518/fix-concurrent-setattributes with baseline commit a3176e0 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 39 metrics, 2 unstable metrics.

[katiehockman]

The changes look good to me, so I'll approve them. However, I made a couple of commits, so I'd like @dianashevchenko to look this over on Monday.

[darccio]

Thanks @katiehockman & @dianashevchenko!