fix(llmobs): fix token extraction for chat completion streams [backport 2.20]

[github-actions]

Backport 75179ef from #12070 to 2.20.

Fixes token chunk extraction to account for the choices field in a chunk being an empty list

Before

Error generating LLMObs span event for span <Span(id=16151817411339450163,trace_id=137677390470467884790869841527646927357,parent_id=None,name=openai.request)>, likely due to malformed span
Traceback (most recent call last):
  File "/XXXXX/ddtrace/contrib/internal/openai/utils.py", line 118, in __aiter__
    await self._extract_token_chunk(chunk)
  File "/XXXXX/ddtrace/contrib/internal/openai/utils.py", line 157, in _extract_token_chunk
    choice = getattr(chunk, "choices", [None])[0]
IndexError: list index out of range

After

Traced succesfully

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: backport-12070-to-2.20
Commit report: 7b3769b
Test service: dd-trace-py

✅ 0 Failed, 130 Passed, 1378 Skipped, 4m 34.45s Total duration (35m 10.66s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-27 06:17:57

Comparing candidate commit 616dec0 in PR branch backport-12070-to-2.20 with baseline commit fe4d74a in branch 2.20.

Found 3 performance improvements and 0 performance regressions! Performance is the same for 391 metrics, 2 unstable metrics.

scenario:iast_aspects-ospathbasename_aspect

• 🟩 execution_time [-365.796ns; -304.094ns] or [-9.814%; -8.158%]

scenario:iast_aspects-ospathjoin_aspect

• 🟩 execution_time [-460.034ns; -393.517ns] or [-8.221%; -7.032%]

scenario:iast_aspects-title_aspect

• 🟩 execution_time [-265.349ns; -226.184ns] or [-8.641%; -7.366%]

[github-actions]

CODEOWNERS have been resolved as:

releasenotes/notes/fix-token-extraction-0133808742374ef4.yaml           @DataDog/apm-python
ddtrace/contrib/internal/openai/utils.py                                @DataDog/ml-observability