Version 2.1.0

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.7 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

What's Changed

• feat(components): Bump image tag used by preview.llm pipelines by @copybara-service in #10295
• feat(components): Add num_microbatches to _implementation.llm training components by @copybara-service in #10248
• chore(sdk): remove cleanup param in local init #localexecution by @connor-mccarthy in #10293
• feat(sdk): add local execution skeleton #localexecution by @connor-mccarthy in #10292
• fix(components): Use llama-2-7b for the base reward model when tuning llama-2-13 with the preview.llm.rlhf_pipeline by @copybara-service in #10249
• feat(sdk): add local task dispatcher validation and ExecutorInput construction #localexecution by @connor-mccarthy in #10298
• test(sdk): install kfp-pipeline-spec from source in kfp sdk tests by @connor-mccarthy in #10300
• fix(sdk): use kfp.dsl.types to replace kfp.components.types Fixes #10282 by @hsinhoyeh in #10283
• feat(components): Enable text generation pipeline to generate row based metrics by @copybara-service in #10296
• test(sdk): add placeholder_utils_test.py by @connor-mccarthy in #10301
• feat(component): Migrate AutoSxS pipeline to preview and move related files to _implementation/llm directory to help Model Eval team use side by side metrics as part of their pipeline by @copybara-service in #10294
• feat(components): Release new LLM Eval image version 0.5 by @copybara-service in #10313
• chore(components): release GCPC 2.7.0 by @copybara-service in #10303
• fix(components): Add autosxs_pipeline to the all variable for the preview/model_evaluation directory by @copybara-service in #10317
• chore(components): release GCPC 2.8.0 by @copybara-service in #10316
• feat(components): Add better docstrings for AutoSxS by @copybara-service in #10320
• test(components): INTERNAL by @copybara-service in #10314
• feat(sdk): add subprocess task handler #localexecution by @connor-mccarthy in #10302
• fix(components): Use large_model_reference as model_reference_name when uploading models from preview.llm.rlhf_pipeline instead of hardcoding value as text-bison@001 by @copybara-service in #10321
• test(sdk): add kfp-kubernetes execution tests by @connor-mccarthy in #10304
• fix(components): Resolve unique model display name on each preview.llm.rlhf_pipeline run instead of reusing cached result by @copybara-service in #10322
• fix(components): Upload the tuned adapter to Model Registry instead of model checkpoint from preview.llm.rlhf_pipeline by @copybara-service in #10323
• feat(sdk): add local execution output collection #localexecution by @connor-mccarthy in #10325
• chore(sdk): fix use of invalid escape sequence in tests by @connor-mccarthy in #10310
• feat(sdk): add local execution logging #localexecution by @connor-mccarthy in #10326
• chore(components): Fix argument description by @copybara-service in #10327
• feat(backend): preserve querystring in pipeline root (fixes #10318) by @TobiasGoerke in #10319
• feat(components): change output format to allow possible post eval by @copybara-service in #10281
• fix(frontend): Add disableParsingRawHTML option for markdown-to-jsx component by @zijianjoy in #10315
• feat(sdk): add DockerRunner #localexecution by @connor-mccarthy in #10328
• feat(sdk): add special dsl.OutputPath read logic #localexecution by @connor-mccarthy in #10334
• chore(sdk): write local execution logs to stdout #localexecution by @connor-mccarthy in #10330
• feat(sdk): support local Container Component execution #localexecution by @connor-mccarthy in #10333
• feat(components): Update RLHF env vars to handle empty string by @copybara-service in #10331
• feat(components): Output errors as a separate table from Arbiter by @copybara-service in #10329
• docs(components): Fix AutoSxS docstring formatting by @copybara-service in #10341
• chore(components): add json_escape placeholder util by @copybara-service in #10351
• fix(sdk): fix presentation of strings in local execution #localexecution by @connor-mccarthy in #10353
• fix(sdk): remove redundant newline character in local DockerRunner logs #localexecution by @connor-mccarthy in #10354
• test(sdk): improve KFP SDK local runner test safety #localexecution by @connor-mccarthy in #10336
• feat(backend): Enable logging for KFP components by @DharmitD in #10288
• feat(sdk): remove local execution feature flag #localexecution by @connor-mccarthy in #10355
• fix(sdk): permit empty local execution outputs #localexecution by @connor-mccarthy in #10338
• feat(sdk): support Concat and IfPresent placeholder in local container component execution #localexecution by @connor-mccarthy in #10348
• test(sdk): add test for local execution of None default parameter #localexecution by @connor-mccarthy in #10339
• chore(sdk): release KFP SDK 2.5.0 by @connor-mccarthy in #10364
• chore(sdk): depend on protobuf 4 in kfp-pipeline-spec by @connor-mccarthy in #10305
• docs(components): Document AutoML Tables util functions by @copybara-service in #10359
• feat(components): Implement the feature store grounding pipeline by @copybara-service in #10332
• feat(components): update eval pipeline documentation to clarify the required pipeline parameters by @copybara-service in #10366
• docs(components): update BigqueryQueryJobOp docs on container args limit by @copybara-service in #10380
• chore(components): GCPC 2.8.1 release by @copybara-service in #10381
• feat(kubernetes_platform): Update kubernetes_platform go package to include pod labels and annotations by @Tomcli in #10357
• chore(sdk): depend on protobuf 4 in kfp and kfp-kubernetes by @connor-mccarthy in #10307
• feat(components): Implement new output format of inference component by @copybara-service in #10375
• chore(components): rollback GCPC 2.8.1 release by @copybara-service in #10385
• chore(sdk): release KFP SDK 2.6.0 by @connor-mccarthy in #10386
• chore(components): Add v1.model.ModelGetOp components by @copybara-service in #10226
• chore(sdk): release kfp-kubernetes 1.1.0 by @connor-mccarthy in #10387
• chore(sdk): update kfp-kubernetes docs versions and release scripts by @connor-mccarthy in #10388
• docs(sdk): fix kfp-kubernetes docs build error by @connor-mccarthy in #10389
• chore(components): Sync AutoML components by @copybara-service in #10372
• fix(components): Update base image for KFP lightweight component for VPC SC compliance by @copybara-service in #10374
• chore: Fix metrics visualization v2 sample by @rimolive in #10399
• docs(sdk): add kfp.local to reference docs #localexecution by @connor-mccarthy in #10395
• chore(components): migrate GCPC to protobuf 4; require KFP>=2.6.0 by @copybara-service in #10401
• feat(components): Support scheduling and labels in utils.build_payload by @copybara-service in #10408
• chore(components): Sync AutoML components by @copybara-service in #10415
• feat(components): support aliases arg in ModelUploadOp by @copybara-service in #10404
• fix(sdk): fix string quotes by @connor-mccarthy in #10413
• docs(components): Update AutoSxS pipeline to use "question_answering" as task name instead of "question_answer", where "question_answer" is still supported, but deprecated by @copybara-service in #10361
• fix(components): Update base image for KFP lightweight component for VPC SC compliance by @copybara-service in #10421
• chore(sdk): local task execution refactor + cleanup #localexecution by @connor-mccarthy in #10420
• fix(backend): MLMD pagination on getting executions of DAG by @hellojunha in #10396
• fix(components): Only run preview.llm.bulk_inference after tuning third-party models with RLHF by @copybara-service in #10425
• feat(components): Update LLM Evaluation Pipelines to use text-bison@002 model by default by @copybara-service in #10428
• chore(components): GCPC 2.9.0 Release by @copybara-service in #10432
• chore(components): fix GCPC documentation dependencies issue by @copybara-service in #10433
• feat(components): Use a single inference component for AutoSxS by @copybara-service in #10424
• feat(sdk): support local pipeline execution #localexecution by @connor-mccarthy in #10423
• fix(components): Write model resource_name to the output of training pipeline remote runner by @copybara-service in #10426
• feat(sdk): support f-strings in local pipeline execution #localexecution by @connor-mccarthy in #10435
• feat(sdk): support for local execution of dsl.importer #localexecution by @connor-mccarthy in #10431
• fix(components): Update text generation pipeline input description by @copybara-service in #10434
• feat(sdk): support local execution of pipelines in pipelines #localexecution by @connor-mccarthy in #10440
• fix(backend): correct run field map col names by @HumairAK in #10430
• chore(components): Create module containing AutoSxS and RLHF image tag by @copybara-service in #10445
• fix(components): Remove row_based_metric file path field in Model Evaluation resource's metadata when row based metrics is disabled by @copybara-service in #10444
• feat(sdk): Support dsl.ParallelFor over list of Artifacts by @KevinGrantLee in #10441
• feat(kubernetes_platform): Update kubernetes_platform go package to include ImagePullSecrets by @JosepSampe in #10410
• feat(kubernetes_platform): Update kubernetes_platform go package to include imagePullPolicy. by @revit13 in #10416
• chore(components): update image tags by @copybara-service in #10455
• fix(sdk): fixes type issues for ParallelFor. Fixes #9366 by @rickyxie0929 in #10436
• feat(components): Add RLAIF pipeline to preview by @copybara-service in #10377
• fix(sdk): fix bug where dsl.OneOf with multiple consumers cannot be compiled by @connor-mccarthy in #10452
• chore(components): Use new module for looking up ReFINED and AutoSxS image tags by @copybara-service in #10454
• feat(Backend + SDK): Update kfp backend and kubernetes sdk to support ImagePullSecrets by @JosepSampe in #10427
• Fix(backend): fix timeout for internal server error. Fixes #10267 by @abaland in #10439
• fix(backend): get pipeline by name is broken due to version typo, Fixes #9940 by @cybernagle in #10268
• feat(kubernetes_platform): Update kubernetes_platform go package to include ConfigMaps as volumes and as env variables. by @roytman in #10400
• feat(Backend + SDK): Update kfp backend and kubernetes sdk to support pod labels and annotations by @Tomcli in #10393
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10472
• feat(kubernetes_platform): Add ActiveDeadlineSeconds(timeout) to the kubernetes platform spec by @Tomcli in #10464
• chore(components): support jinja 3 in GCPC by @copybara-service in #10479
• test(components): Reduce sagemaker component test flakiness by @ananth102 in #10225
• chore(sdk): release KFP SDK 2.7.0 by @connor-mccarthy in #10461
• fix(components): Use PipelineJob location in AutoSxS components, add init file by @copybara-service in #10476
• feat(kubernetes_platform): Add k8s FieldPath as env to the kubernetes_platform by @Tomcli in #10485
• chore: Add Tomcli as a backend approver by @chensun in #10490
• chore(components): Sync AutoML components by @copybara-service in #10492
• chore(README): Update Kubeflow Pipelines on Tekton blog by @Tomcli in #10482
• chore(components): Rename several _implementation.llm components by @copybara-service in #10474
• feat(kubernetes_platform): Update kubernetes_platform go package to i… by @droctothorpe in #10442
• docs(components): internal by @copybara-service in #10495
• chore(components): bump highest support KFP SDK version in GCPC to KFP SDK 2.7.0 by @copybara-service in #10498
• fix(backend): fixes "cannot save parameter" error message. Fixes #9678 by @hbelmiro in #10459
• fix(rlhf): Supporting adapter only output for reward model training by @copybara-service in #10503
• chore(components): internal by @copybara-service in #10511
• docs(components): internal by @copybara-service in #10505
• feat(components): Added experimental args to batch_prediction_pairwise component by @copybara-service in #10497
• chore(components): release GCPC SDK 2.10.0 by @copybara-service in #10516
• feat(Backend + SDK): Update kfp backend and kubernetes sdk to support tolerations by @droctothorpe in #10471
• feat(components): Add configurable image prefix to llm utility method by @copybara-service in #10513
• chore(components): internal change by @copybara-service in #10512
• feat(components): Add CMEK support to preview.llm.rlhf_pipeline by @copybara-service in #10456
• feat(Backend + SDK): Update kfp backend and kubernetes sdk to support ConfigMaps as volumes and as env variables by @roytman in #10483
• feat(components): Release Forecasting training pipelines to V1 namespace by @copybara-service in #10523
• fix(sdk): Prevents dsl.ParallelFor over single parameter from compiling. by @KevinGrantLee in #10494
• fix(components): Pass tuned model checkpoint to inference pipeline after RLHF tuning by @copybara-service in #10518
• test: Add ParallelFor compile test over single artifact. by @KevinGrantLee in #10531
• fix(components): rename custom task calibration_score_rubric -> score_rubric by @copybara-service in #10525
• chore(components): Switch default machine type for LLM Text Generation Eval pipeline and components to e2-standard-4 by @copybara-service in #10522
• fix(components): Propagate location to sub-components in AutoSxS by @copybara-service in #10514
• feat(backend + SDK): Add backend and SDK support to use Kubernetes FieldPath as env by @Tomcli in #10496
• chore(components): Change docker image URI used by preview.llm pipelines by @copybara-service in #10536
• fix(docs): Updated legal info due to migration from CLA to DCO by @hbelmiro in #10501
• feat(backend + SDK): Add Backend and SDK support for timeout in pod spec by @Tomcli in #10481
• fix(components): Return None as sliced feature attribution values for the classes which are not predicted in bp outputs by @copybara-service in #10539
• feat(components): Add CMEK validation to preview.llm.infer_pipeline by @copybara-service in #10519
• feat(Backend + SDK): Update kfp backend and kubernetes sdk to support ImagePullPolicy by @revit13 in #10417
• docs(components): fix create_custom_training_job_from_component docs rendering by @copybara-service in #10541
• feat(components): Implement the train time evaluation in reward model training. With the train time eval dataset available, the pipeline outputs the accuracy and cross entropy metrics to the log by @copybara-service in #10369
• fix(samples): Updated samples/core to V2 by @PratyushaRavi in #9879
• chore(components): Consolidate validation components in preview.llm.rlhf_pipeline by @copybara-service in #10543
• feat(backend): Upgrade go version to 1.20 by @Tomcli in #10502
• docs(components): Modify the GetModel documentation by @copybara-service in #10552
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10551
• fix(backend): Fixes response status of http error code when uploading duplicate pipeline [Fixes #10311] by @champon1020 in #10546
• chore: Change stalebot rules by @rimolive in #10547
• fix(kubernetes_platform): Add optional field to SecretAsVolume and ConfigMapAsVolume. Fixes #10548 by @revit13 in #10549
• feat(components): Add CMEK support to AutoSxS pipeline by @copybara-service in #10555
• feat(components): Update _LLM_EVAL_VERSION to v0.6 by @copybara-service in #10565
• chore(components): update container image of endpoint batch predict component for vulnerability patch by @copybara-service in #10566
• chore(backend): Update kfp driver and launcher images by @chensun in #10561
• fix(components): Add relevant component and pipeline inputs/outputs to support creating ModelEvaluations as part of the AutoSxS Metrics component by @copybara-service in #10554
• feat(components): Release Forecasting training pipelines to V1 namespace by @copybara-service in #10569
• feat(components): Add location validation to preview.llm.rlhf_pipeline by @copybara-service in #10542
• chore(components): Testing fix by @copybara-service in #10576
• fix(samples): Update resource_spec, retry, secret samples to v2 pipelines by @junggil in #9876
• fix: Modified the comment/text for pipeline_version_id by @amadhusu in #10581
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10582
• chore: Update api-generator to use golang 1.20 by @chensun in #10580
• fix(docs): make full version dropdown show on all KFP SDK docs versions by @connor-mccarthy in #10577
• feat(components): Implement new component to preprocess and validate inputs for rlhf by @copybara-service in #10480
• chore(components): GCPC 2.11.0 Release by @copybara-service in #10573
• fix(components): Fix missing pipeline parameters by @copybara-service in #10587
• chore: Update release image by @chensun in #10599
• fix: Modified the swagger json files according to the modified proto files. by @amadhusu in #10591
• feat(components): Copy text generation eval and text classification evak pipelines from preview to v1 by @copybara-service in #10586
• docs(components): internal by @copybara-service in #10556

New Contributors

• @TobiasGoerke made their first contribution in #10319
• @rimolive made their first contribution in #10399
• @hellojunha made their first contribution in #10396
• @HumairAK made their first contribution in #10430
• @KevinGrantLee made their first contribution in #10441
• @JosepSampe made their first contribution in #10410
• @revit13 made their first contribution in #10416
• @cybernagle made their first contribution in #10268
• @roytman made their first contribution in #10400
• @hbelmiro made their first contribution in #10459
• @PratyushaRavi made their first contribution in #9879
• @champon1020 made their first contribution in #10546
• @amadhusu made their first contribution in #10581

Full Changelog: 2.0.5...2.1.0