Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gcp_pubsub output: not recovering from failures when ordering key is enabled #2935

Open
erdody opened this issue Oct 15, 2024 · 0 comments
Open
Labels
bug needs investigation It looks as though have all the information needed but investigation is required

Comments

@erdody
Copy link

erdody commented Oct 15, 2024

Scenario

A gcp_pubsub connector (v4.37.0) configured something like this:

input:
  kafka: # btw, this is why we can't move to kafka_franz: https://github.com/redpanda-data/connect/issues/2745
    addresses: [ localhost:19092 ]
    topics: [ ordered-topic ]
    consumer_group: ordered-cg
    checkpoint_limit: 1 # https://docs.redpanda.com/redpanda-connect/components/inputs/kafka/#ordering
    batching:
      count: 100
      period: 1s
output:
  gcp_pubsub:
    project: <redacted>
    credentials: <redacted>
    topic: ordered-topic
    ordering_key: ordering-key

Expectation

After, for example, an internet interruption, the connector should automatically recover and resume publishing.

What we see

The connector does not auto-recover, and keeps failing with:

{"@service":"redpanda-connect","label":"","level":"error","msg":"Failed to send message to gcp_pubsub: pubsub: Publishing for ordering key, ordering-key, paused due to previous error. Call topic.ResumePublish(orderingKey) before resuming publishing","path":"root.output","stream":"stream-pubsub"}

you have to restart the process to restore the flow.

It looks like a call to topic.ResumePublish is missing, either after a failure or before each batch.

@mihaitodor mihaitodor added bug needs investigation It looks as though have all the information needed but investigation is required labels Oct 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug needs investigation It looks as though have all the information needed but investigation is required
Projects
None yet
Development

No branches or pull requests

2 participants