ADR - Options to prevent bugzilla from pausing webhook calls

[alexcottner]

Added ADR with proposed options to prevent bugzilla from pausing webhook calls due to configuration errors.

Link to ADR for easy reference.

Issues this could solve (or affect) depending on solution we implement:

• System is able to "drop" erring messages without user interference #710
• Create alerts for when bugs fail to sync #743
• Establish convention for capturing system incidents #730

[bsieber-mozilla]

Thanks for this ADR @alexcottner ! This is certainly an issue worthy of an ADR. In terms of options--I think option 1, possibly with some slight addition would be a good approach.

In terms of options 3, 4, 5: I'm hesitant for adding additional infra, like a redis-cache/queue, when there's already a "queue" in the system (the bugzilla webhook queue). I understand it would give JBI more control over the messages, but also is data redundant.

In terms of option 1, depending on how we log, we could use those logs to automate alerts (https://cloud.google.com/logging/docs/alerting/log-based-alerts) which could then be sent to slack/email/other channels.

That way we "log and move-on" but also have a breadcrumb trail in public space so our users could be aware that they might be affected by data-loss. The alert could reveal the particular configuration, and bug_id so that interested parties could jump if they were expecting something that did not appear.

Thanks again for spinning up this ADR! 🙇

[leplatrem]

See #82

One main comment: the format would require a bit more structure to follow the arguments better (criteria/decision drivers in front of each proposed solution). See other ADRs in docs/adrs/

Also, there is another solution that Graham suggested a while ago: when onboarding a project on JBI, we assign a stakeholder that is in charge of setting up the webhook for their product in Bugzilla. When there is an error, this stakeholder is alerted, and is charge of fixing it to unblock their queue.

[leplatrem]

About option 1: log and move on. How long do we retain logs for?

What about a solution where we log the error (+notification to devs on Slack) and have a super simple DLQ?
For example, we would append failing webhook payloads to a file, and have a command to replay them and purge the file? Payloads failing again during retry would be added to DLQ of the current day/month.
If file-based sounds too prehistoric and not flexible enough for statistics and stuff, we could have a simple bigquery table, that unlike logs could be kept forever.

[alexcottner]

About option 1: log and move on. How long do we retain logs for?

Currently this is 30 days in log-explorer and 90 days in bigquery. But we could do this for as long or as little as we wanted, or even stream these specific logs to a bucket as a DLQ.

What about a solution where we log the error (+notification to devs on Slack) and have a super simple DLQ? For example, we would append failing webhook payloads to a file, and have a command to replay them and purge the file? Payloads failing again during retry would be added to DLQ of the current day/month. If file-based sounds too prehistoric and not flexible enough for statistics and stuff, we could have a simple bigquery table, that unlike logs could be kept forever.

Appending to single a file is annoying when running parallel containers. But if the spirit of this is "send it to a DLQ that we could re-process later", then it's easy enough to send them to a bucket or a volume on K8s. It would probably be something like /{year}/{month}/{date}/{pod-id}-{message-id}.json.

[alexcottner]

We had a discussion today where I think we're leaning towards one or both of these:

• Option 2, ask a human to do something
• Option 4, use a simple DLQ

Both of these are relatively small efforts to implement with good outcomes. If we can easily generate alerts based on our simple DLQ (ex: an emailed report from a postgres table or messaging an IM channel based on a file being dropped into a bucket) then this could be a simple solution with high resiliency.

[leplatrem]

Both of these are relatively small efforts to implement with good outcomes. If we can easily generate alerts based on our simple DLQ (ex: an emailed report from a postgres table or messaging an IM channel based on a file being dropped into a bucket) then this could be a simple solution with high resiliency.

💯

I vote to start with Option 4 with files on disk (mounted volume), and iterate if needed

And with regards to alerting, we could have another dedicated ADR (bugzilla comment vs Slack vs email to workflow owner, etc.)

[grahamalama]

I think we're underestimating the engineering effort of Option 4. By "engineering effort" here I mean code complexity, not necessarily infrastructure.

We would always return 200, but any events that fail to process internally would get sent to a DLQ and be replayed later if needed. ... A scheduled kubernetes job would then run to try and pick these up and reprocess them later (every 4 hours, for exmaple).

There's a lot packed in here. How will we decide when it's needed (or not needed) to replay a message from the DLQ? How will we avoid situations where, for example:

• user creates a bug, it syncs
• user adds EDIT 1: ... to the summary
• some network error happens, the bug fails to sync, and it's placed in the DLQ
• the network error solves itself
• user adds EDIT 2: ... to the summary. The summary is synced.
• 4 hours later, we replay the message in the DLQ. The summary is synced, but it erases EDIT 2 since it wasn't on the message in the DLQ

Also, something I saw on AWS docs on DLQs:

Don't use a dead-letter queue with a FIFO queue if you don't want to break the exact order of messages or operations.

I think for now, that's us (illustrated by the example above). I think we could code our way around it in certain cases, but that adds to the complexity of this option

[grahamalama]

Also, something I noticed in the ADR:

Option 2:
- Amount of initial engineering effort: low
- Amount of maintenance effort: low

Option 4:
- Amount of initial engineering effort: mid
- Amount of maintenance effort: mid-low

Based on our own criteria, Option 2 requires lower effort and maintenance that Option 4. Performance and intuitiveness are the same. What then, is compelling us to choose Option 4? Are there some other considerations that we haven't documented in the ADR that make Option 4 more attractive?

[alexcottner]

Good points, processing data out of order can be just as dropping it. Gonna update the ADR with that as a risk for now. And since we're stateless we can't compare timestamps easily.

[leplatrem]

Some feedback after re-reading again the whole doc...

I haven't fully understood how you would distinguish from blocking events from others. Is it based on Jira HTTP responses? (ie. 401/500 vs. 400)

How do we merge events? What is the cardinality? (product, bug #, event_type) ?

I know that this ADR is about choosing an overall strategy and not going into implementation details, but I believe that the chosen solution would be easier to understand if some aspects were detailed.

[leplatrem]

I read again quickly, seems to match our implementation.

I think the format of this ADR is not exactly like others we have, where we list all options, and in the Chosen solution we only say which one we chose and why.

Examples:

• https://github.com/mozilla/remote-settings/blob/main/docs/adr/adr_002_packaging_releasing_admin_UI.md#decision-outcome
• https://github.com/mozilla-services/telescope/blob/main/docs/adrs/adr_0001_history.md#decision-outcome

But this may not be very important :)