Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate CSV tools support files for supported Databricks versions #10448

Open
jlowe opened this issue Feb 20, 2024 · 1 comment
Open

Generate CSV tools support files for supported Databricks versions #10448

jlowe opened this issue Feb 20, 2024 · 1 comment
Labels
build Related to CI / CD or cleanly building task Work required that improves the product but is not user facing tools

Comments

@jlowe
Copy link
Member

jlowe commented Feb 20, 2024

Relates to #10440. There should be regular generation of tools support CSV files for supported Databricks platforms.

Steps:

  • Update tools/pom.xml so it can generate a proper classpath to run the tools generator code on Databricks. The only thing tricky here is getting the classpath right, because in Maven it doesn't really understand how to build a full classpath to run Spark, just enough to get compiles against it. Honestly it might be simplest to hack the classpath so we pick up Databricks jars via -cp /databricks/jars/* when running this.
  • Update jenkins/databricks/build.sh so it builds the verify target rather than the package target (so it generates the tools files)
  • Modify the databricks premerge and nightly CI so builds will fail if tools files are found to be checked out after a build. Likely need to loop in CI/CD team for this.
@jlowe jlowe added ? - Needs Triage Need team to review and classify task Work required that improves the product but is not user facing tools labels Feb 20, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Feb 27, 2024
@mythrocks mythrocks added the build Related to CI / CD or cleanly building label Jan 8, 2025
@mythrocks
Copy link
Collaborator

cc @sameerz, @GaryShen2008. Based on discussion with the team, this would need collaboration between the Build team and the Tools team.

The Tools team (?) would need to update the generator script to work on Databricks environments. The Build team could then invoke that during mvn verify to generate the supportedOps csv files, for the build artifact.

I don't know of the priority to include this in 25.02.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building task Work required that improves the product but is not user facing tools
Projects
None yet
Development

No branches or pull requests

3 participants