Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion: drive data emission with proctime/now when there's no new data in EOWC queries #19860

Open
stdrc opened this issue Dec 18, 2024 · 0 comments

Comments

@stdrc
Copy link
Member

stdrc commented Dec 18, 2024

In emit-on-window-close queries, when there's no new data (hence new watermarks) coming, some data can be pending forever inside the internal states of RW, not being emitted.

Some user want to emit these pending data after a period of time even when no new data come in.

A possible solution can be (just a prelimilary thought, still need discussion):

CREATE SOURCE s (
  ...
  event_time TIMESTAMPTZ
  WATERMARK FOR event_time AS (event_time - interval '5 minutes', now() - interval '10 minutes')
)

So that when there's no new event_time, now() - interval '10 minutes' can be used as watermark expression.

@github-actions github-actions bot added this to the release-2.2 milestone Dec 18, 2024
@stdrc stdrc removed this from the release-2.2 milestone Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant