feat(kfp): restructure pipeline to allow mocking sections

[tumido]

This PR restructures our approach to KFP pipelines. It also adds a CLI interface:

python pipeline.py --help    
Usage: pipeline.py [OPTIONS]

Options:
  --mock [sdg|train|eval]  Mock part of the pipeline
  --help                   Show this message and exit

Since we can't pass data between nested pipelines (kubeflow/pipelines#10041), we need to use a flat, single pipeline.

Each stage code components live in their respective folders as a python package:

sdg
 ├── __init__.py        // Package for the stage 
 ├── componenets.py     // Individual components
 └── faked
      ├── __init__.py        // Package for the faked stage
      └── componenets.py     // Individual faked components

In order to provide better dev experience, each stage can be mocked individually. This is done via component substitution - it preserves the component signature but replaces the body. This way developers can mock individual outputs (where it matters) so we don't break continuity.

In this example I provide mocked components for the SDG stage:

if 'sdg' in mock:
    from sdg.faked import git_clone_op, sdg_op
else:
    from sdg import git_clone_op, sdg_op

In order to provide output artifact from SDG, I've used a trick for Python lightweight components (yes, it's a hack). This trick bypasses the "Hermetic" nature of KFP Python lightweight components. I'm creating a new empty shell python package that provides data via setuptools.data-files. This package is installed on the fly to the component runtime. The faked component then ensures that the data gets copied to the output artifact.

I order to test the pipeline with mocked SDG do:

1. Replace package url at main...tumido:kfp-to-cli?expand=1#diff-8c5a3a6ccb with:

   git+https://github.com/tumido/ilab-on-ocp.git@kfp-to-cli#subdirectory=sdg/faked/fixtures
   

2. Run

   python pipeline.py --mock sdg
   

3. This will update pipeline.yaml and replace each SDG step content with a faked component. Upload the new pipeline.yaml to KFP and run.
4. Once executed, you get the same execution flow as if it was not faked - but the steps actually do not do the work. The API and component signatures are maintained, but no heavy lifting is happening. 🙂

[boarder7395]

@tumido out of curiousity why not just fix the bug that prevents nested pipelines from passing artifacts? I ask because I plan to start looking at that issue and am wondering if you already found some unsurmountable issue which makes the hack approach a better option?

[tumido]

@boarder7395 At this moment I'm glad I know KFP good enough from the user perspective. It would take me a while to obtain skills in working on KFP itself. It's a codebase I never touched. Besides that I need to have the pipeline running on our current KFP deployment. I can't be waiting at this moment for a PR getting merged into KFP, then bubbling through KFP release process, then getting the new KFP version adopted by RHOIA, then getting RHOAI updated on our cluster. It's a lot faster to workaround the issue.

Especially when my job right now is to get the pipeline working and I can't justify weeks of studying of how KFP works internally, how to set up a dev environment, how to write tests, etc... all of it. I know... in ideal world it would be nice to be able to always resolve the root cause. 🤷

Besides, this is not much of a hack - instead of having a master pipeline and then multiple pipelines per stage, that would be chained from the umbrella master pipeline, we just define a single pipeline that runs all the steps. The mocking would need to happen in either case - we want to be able to develop the stages independently but stages depend on data from previous stage - this data need to be provided by something somehow. Fixing the issue above would change nothing here.

[boarder7395]

@tumido That makes sense to me, the one looking down the barrel of doing exactly those steps now. I was familiar with kfp 1.0 but 2.0 has been something I've avoided. Just wanted to make sure there wasn't already a dead end at the end of the tunnel :)

[MichaelClifford]

LGTM 🚀

[tumido]

Then I'm gonna self-merge. Because I can. 😄 🙌