Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate with datafusion to support filters pushdown from SQL #160

Closed
xushiyan opened this issue Oct 8, 2024 · 4 comments · Fixed by #203 or #221
Closed

Integrate with datafusion to support filters pushdown from SQL #160

xushiyan opened this issue Oct 8, 2024 · 4 comments · Fixed by #203 or #221
Assignees
@xushiyan @jonathanc-n
Labels
feature
Milestone
release-0.3.0

Comments

@xushiyan
Copy link
Member

xushiyan commented Oct 8, 2024

Follow-up feature of #47

Integrate with datafusion on the predicate expression parsing and support pushdown predicates from SQL.
@xushiyan xushiyan added this to the release-0.4.0 milestone Oct 8, 2024
@xushiyan xushiyan mentioned this issue Oct 8, 2024
Closed
@jonathanc-n
Copy link
Contributor

May I be assigned this?

@xushiyan
Copy link
Member Author

@jonathanc-n sure. can you first elaborate on the changes you're gonna make? like what structs and APIs you'll introduce and some high level design to show how it works for the requirement?

@jonathanc-n
Copy link
Contributor

Probably will involve these:

  • Convert DataFusion's Expr objects into Hudi filters added in feat: support partition prune api #119 .
  • Implement the supports_filters_pushdown method in the TableProvider.
  • Change scan method to pass translated filters to Hudi's apis

@xushiyan
Copy link
Member Author

@jonathanc-n sounds good

@xushiyan xushiyan modified the milestones: release-0.4.0, release-0.3.0 Nov 27, 2024
@jonathanc-n jonathanc-n mentioned this issue Nov 29, 2024
Merged
@xushiyan xushiyan mentioned this issue Nov 30, 2024
Open
@xushiyan xushiyan moved this from Todo to In Progress in hudi-rs roadmap Nov 30, 2024
This was referenced Nov 30, 2024
Open
Closed
@github-project-automation github-project-automation bot moved this from In Progress to Done in hudi-rs roadmap Dec 8, 2024
@xushiyan xushiyan self-assigned this Dec 8, 2024
@xushiyan xushiyan mentioned this issue Dec 8, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xushiyan xushiyan

@jonathanc-n jonathanc-n

Labels
feature
Projects
hudi-rs roadmap
Archived in project
Milestone
release-0.3.0
2 participants
@xushiyan @jonathanc-n