[APM][Spans] Instrument edxapp celery workers in Datadog

[robrap]

See https://ddtrace.readthedocs.io/en/stable/integrations.html#celery.

I also asked in tech-external-datadog.

• Required for ATAM decision: No (Assuming it is possible)
• Required for contract pricing: Yes (To see load)
• Required for completing migration: Yes

AC:

• Instrument celery for workers (and potentially task producers)

[robrap]

It's unclear whether the instrumentation is both for firing tasks (webapps) and consuming tasks (workers), or just for the workers.

[connorhaugh]

1. Check to see if arch-experiments is still installed/deployed. List all ides where celery workers are used.
2. Figure out where we NEED to be pluggable, and where we don't need to be pluggable.
3. Maybe use arch-experiments as a point for plugin-requiring code.
4. Propose a plan to axim for implementation details, get buy in.

[connorhaugh]

5. determine in NR if kafka is a "big enough thing" to matter now, and maybe do it alongside celery if it is trivial.

[connorhaugh]

arch-experiments is still installed/deployed.

Yes. https://github.com/edx/edx-internal/blob/66d1c10e06e685369bf00947870073625f0fc6a4/ansible/vars/prod-edx.yml#L317

List all IDEs where celery workers are used.

• Edx-Platform
• Ecommerce (ecommerce-worker)
• Credentials Service
• Enterprise Catalog
• Course Discovery
• Enterprise Access
• Registrar

[connorhaugh]

It's unclear whether the instrumentation is both for firing tasks (webapps) and consuming tasks (workers), or just for the workers.

From some digging, it seems that there is no clear documentation on what occurs. I think from what I see in the data the firing tasks are cataloged, I don't really know exactly what denotes a firing task though.

I was able to create logs for celery tasks for edx-platform in devstack, which is proof of concept enough for the other services (it took a while to set up local parallel celery on edx-platform and that is already a paved path).

Here are the PRs for the required services, followed by the revert PR.

edx platform:
openedx/edx-platform#34537

openedx/enterprise-catalog#817

openedx-unsupported/registrar#612

[connorhaugh]

openedx/course-discovery#4335
openedx-unsupported/ecommerce#4155

[robrap]

@connorhaugh: Do we have a long-term plan, or were we waiting until after the experiment for this? I not well thought through plan might be:

1. Introduce a setting for a pre-celery config hook.
2. Introduce something like 2u-django-utils that can house code similar to edx-django-utils, but that we privately plug in to all our services, that would implement the pre-celery hooks api.

[robrap]

@timmc-edx has a PR that probably needs to land first, which may relate to the longer term version of this work.

[connorhaugh]

openedx/edx-django-utils#409

[timmc-edx]

@connorhaugh This can be closed out now, right? (Along with the PR in the previous comment.)

[robrap]

@timmc-edx: FYI: I'm closing this Issue, and I closed the above PR. I assume at this point we do not need this.