Thoughts on Gatsby + Observability

[KyleAMathews]

A braindump of some things I've been thinking about.

We have tracing in the framework but it has a few problems:

• we used https://opentracing.io/ but that's now been deprecated in favor of https://opentelemetry.io/ — there's a shim to make the two work together (see fix(gatsby): fix nesting of tracing spans + add docs for OpenTelemetry #33098) — but at some point we'll want to migrate
• OpenTelemetry would let us add metrics support which would be helpful for isolating memory leaks especially
• we've mingled activities and telemetry spans together in a way I don't think is ideal. Activities are UI focused (as they show up in the CLI and other outputs). Some activities are too high-level and offbeat for tracing and some are far too low-level to be of interest to 99% of users. Activities should just show at a high-level what's happening + errors and tracing should be very verbose for deep analysis.
• spans aren't always named consistently — we could create patterns for names for core/plugins & create a package (or add to gatsby-core-utils) of common names similar to https://www.npmjs.com/package/@opentelemetry/semantic-conventions
• we need patterns for tracing plugins — see my PR for gatsby-source-drupal as a stab at this feat(gatsby-source-drupal): Add tracing for full/delta fetches and http requests #33142
• Only ~50% of the framework is traced
• we don't trace graphql by default as it impacts perf. But graphql is also one of the most useful things to trace. Let's take a sampling approach to tracing grapqhql to balance perf and detailed telemetry.

Moving to OpenTelemetry has some additional interesting advantages in that they're working to standardize APIs & SDKs for backends for the grand trinity of observability — tracing, metrics, and logging. They haven't yet shipped support for metrics/logging (their roadmap doc though is 💯 but when they do, having a standard way in the framework/plugins to output tracing/metrics/logs that interoperate with a wide variety of backends would be wonderful.

[MarkBennett]

One other area to add to this is support for telemetry in Gatsby Functions. We're looking at adding this ourselves now, so that we can push logs to Sentry and Google Cloud Tracing, but it would be helpful if there was official support for this.