Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not handle NotIn guarantees in the BufferFilter #25881

Open
hiltontj opened this issue Jan 21, 2025 · 0 comments
Open

Do not handle NotIn guarantees in the BufferFilter #25881

hiltontj opened this issue Jan 21, 2025 · 0 comments
Assignees
Labels

Comments

@hiltontj
Copy link
Contributor

Problem statement

The BufferFilter and downstream code that uses it when indexing the buffer could be simplified by removing the handling of NotIn guarantees.

They provide little value, expecially if they just end up selecting most of what is in the buffer - at that point, we might as well just let DataFusion do the filtering.

Proposed solution

Skip NotIn guarantees when processing the results of LiteralGuarantee::analyze in the BufferFilter::new method:

let literal_guarantees = LiteralGuarantee::analyze(&physical_expr);
for LiteralGuarantee {
column,
guarantee,
literals,
} in literal_guarantees
{

This will break some tests, so the tests in table_buffer.rs will need to be cleaned up, as sell as the logic that is handling the NotIn guarantees.

Additional context

See #25866 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant