Move EnforceSorting into datafusion-physical-optimizer crate

[alamb]

Is your feature request related to a problem or challenge?

• Part of [EPIC] Extract remaining physical optimizer out of core #11502
• Related to Building project takes a *long* time (esp compilation time for datafusion core crate) #13814

Historically DataFusion was one (very) large crate datafusion, and as it grew bigger we extracted various functionality into separate crates. This leads to both faster compile times (as the crates can be compiled in parallel) as well easier to navigate code (as the crates force a cleaner dependency separation)

One project we have not yet completed is to extract the physical optimizer passes out #11502

• The original physical optimizers are here https://github.com/apache/datafusion/tree/main/datafusion/core/src/physical_optimizer
• The new crate is here https://github.com/apache/datafusion/tree/main/datafusion/physical-optimizer

Describe the solution you'd like

Extract the ProjectionPushdown from the datafusion core crate to the datafusion-physical-optimizer crate

Describe alternatives you've considered

Move the code

• From https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_optimizer/enforce_sorting.rs
• To https://github.com/apache/datafusion/tree/main/datafusion/physical-optimizer/src/

I believe you will have to move two other modules:

• https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_optimizer/replace_with_order_preserving_variants.rs
• https://github.com/apache/datafusion/blob/main/datafusion/core/src/physical_optimizer/sort_pushdown.rs

I would recomend the following structure:

datafusion/physical-optimizer/src/enforce_sorting/mod.rs (old enforce_sorting.rs)
datafusion/physical-optimizer/src/enforce_sorting/replace_with_order_preserving_variants.rs 
datafusion/physical-optimizer/src/enforce_sorting/sort_pushdown.rs 

Notes that due to dependencies (e.g. on SessionContext or functions), you may have to move the tests into the core_integration tests here:

• https://github.com/apache/datafusion/tree/main/datafusion/core/tests/physical_optimizer

Additional context

Here are some example PRs

• chore: move SanityChecker into physical-optimizer crate #14083
• Move JoinSelection into datafusion-physical-optimizer crate (#14073) #14085
• Move Pruning into physical-optimizer crate #13485

I think this is likely not a good first issue as it will invovle substantial code

[buraksenn]

take

[logan-keede]

oops, We have a PR clash. @buraksenn Is your PR complete? I can try and merge it into mine, or the other way around.

[buraksenn]

@logan-keede I just saw this comment. I had a test error that I'm fixing now. I think it will be complete in an hour

[logan-keede]

@buraksenn can you push it directly to #14190, I have already merged your previous commits so it should not cause you much trouble?
Thanks

[buraksenn]

Sorry I could not look into this afterwards and currently working. I think my PR is ready for merge so maybe we can merge that one and then rebease from main. Otherwise, I can look into this tomorrow

[logan-keede]

No worries, I did it myself.

[alamb]

Sorry I could not look into this afterwards and currently working. I think my PR is ready for merge so maybe we can merge that one and then rebease from main. Otherwise, I can look into this tomorrow

Thanks everyone -- I just approved the following PR and hopefully we can merge it tomorrow

• Move EnforceSorting into datafusion-physical-optimizer crate  #14219

[berkaysynnada]

I am taking a final look on #14219, and #14190, and I'll merge them if not see a problem.