KFP SDK v2.10.0

Release of the KFP SDK only.

To install the KFP SDK:

pip install kfp==2.10.0

For changelog, see release notes.

KFP SDK v2.9.0

Release of the KFP SDK only.

To install the KFP SDK:

pip install kfp==2.9.0

For changelog, see release notes.

Version 2.3.0

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.7 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

What's Changed

• chore: Update RELEASE.md by @chensun in #10767
• fix(components): Add input param autorater_prompt_parameters to online_evaluation_pairwise component by @copybara-service in #10759
• feat(components): create infer preprocessor component by @copybara-service in #10765
• feat(components): create utility class for preprocessors and use it in rlhf preprocessor and infer preprocessor by @copybara-service in #10768
• chore(sdk): drop Python 3.7 for kfp sdk by @rickyxie0929 in #10750
• chore: Promote @rimolive as a backend approver by @chensun in #10780
• feat(components): use preprocessor utility methods for the upload model graph by @copybara-service in #10774
• chore: upgrade node to v18 by @HumairAK in #10794
• feat(GH workflow): migrate periodic functional tests to GH actions by @shruti2522 in #10751
• chore: Add Container Engine and Image Tag parameters to backend Makefiles by @gmfrasca in #10725
• chore: add meaningful logs to client initialization. by @HumairAK in #10801
• feat(kubernetes_platform): Update kubernetes_platform go package to include EnabledSharedMemory by @hsteude in #10703
• feat(components): Retry on batch prediction internal errors in AutoSxS by @copybara-service in #10799
• fix(components): Add staging and temp locations to prophet trainer component by @copybara-service in #10803
• fix(components): Remove unused import function_based from infer pipeline by @copybara-service in #10802
• docs(components): Bump the python version to 3.8 by @copybara-service in #10764
• chore(components): GCPC 2.14.1 Release by @copybara-service in #10813
• chore(components): Rollback GCPC 2.14.1 Release due to b/339911077 by @copybara-service in #10818
• chore(components): Add TaskError proto by @copybara-service in #10823
• chore(components): GCPC 2.14.1 Release by @copybara-service in #10821
• feat(components): Create the write_user_defined_error function by @copybara-service in #10835
• test: fix CI failure for periodic functional tests by @tmvfb in #10817
• chore(docs): Fix link by @haoxins in #7533
• chore: Update Periodic Functional Tests to run once a day by @DharmitD in #10842
• test: fix version conflict for functional tests by @tmvfb in #10837
• chore: server should skip sample load if path DNE by @DharmitD in #10833
• fix(frontend): reduce list run latency by @droctothorpe in #10797
• feat(components): Support parsing Gemini BP outputs in AutoSxS pipeline by @copybara-service in #10853
• feat(components): Use GetModel integration test to manually test write_user_defined_error function by @copybara-service in #10843
• docs: modified the ConfigMap as Env variable example in README.md with right key_name within use_config_map_as_env() block. by @vamsi-01 in #10855
• fix(sdk): Add required auth scopes to RegistryClient for GCP service accounts credentials by @PChambino in #10819
• chore(deps): bump requests from 2.31.0 to 2.32.2 in /backend/src/apiserver/visualization by @dependabot in #10858
• chore: Change command to apply manifests in the docs by @rimolive in #10864
• fix(manifests): Move metacontroller to the top in kustmization.yaml by @codablock in #10669
• test: Migrate Backend tests to a GHA workflow by @DharmitD in #10871
• feat(backend): mount EmptyDir volumes for launcher write locations by @HumairAK in #10857
• docs(components): internal by @copybara-service in #10874
• feat(components): Add role_field_name and model_name as input parameters to llm_evaluation_preprocessor component to support gemini model's input and output schema by @copybara-service in #10875
• fix(components): add check and add log to call out the fallback to the default model checkpoint and remove the model checkpoint check condition in RLHF GCPC by @copybara-service in #10828
• test: migrate kubeflow-pipeline-e2e-test to GitHub Actions by @hbelmiro in #10887
• feat(components): Support dynamic machine type paramters in CustomTrainingJobOp by @KevinGrantLee in #10883
• fix(kubernetes_platform): fix api-generator docker mount for SELinux by @gregsheremeta in #10890
• feat(components): Add Starry Net forecasting pipeline to public preview by @copybara-service in #10888
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10906
• feat(components): Update Starry Net image tags by @copybara-service in #10920
• chore: Updated Python image to 3.8 in backend/Dockerfile by @hbelmiro in #10941
• Expose starry_net yaml to GitHub by @chensun in #10943
• chore(sdk): release KFP SDK 2.8.0 by @chensun in #10940
• chore: Upgrade Python version to 3.8 in readthedocs config by @chensun in #10945
• feat(frontend&backend): Add UI support for object store customization and prefixes by @HumairAK in #10787
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10958
• chore(components): GCPC 2.15.0 Release by @copybara-service in #10951
• docs(backend): Remove deprecated v2 compatibility mode docs by @droctothorpe in #10956
• Upgrade go version to 1.21 by @DharmitD in #10911
• chore: Add ability to mount self-signed certs to kfp by @DharmitD in #10849
• feat(components): add persistent_resource_id to v1 GCPC custom job components/utils by @copybara-service in #10954
• chore: Added @rimolive, @hbelmiro and @DharmitD as approvers and reviewers for GitHub Actions by @hbelmiro in #10963
• test: Migrate KFP SDK YAPF test to a GHA Workflow by @DharmitD in #10961
• test: Created a new GitHub action to reuse the creation of a KFP cluster by @hbelmiro in #10946
• test: kubeflow-pipeline-upgrade-test migrated to GitHub Actions by @hbelmiro in #10932
• test: Migrate Frontend tests to a GHA Workflow by @DharmitD in #10972
• test: install kfp on github periodic functional tests workflow by @tmvfb in #10859
• chore: Add an option to re-run GHA workflows by @rimolive in #10930
• chore(components): Update Starry Net description and add nan- and zero-threshold args by @copybara-service in #10986
• test: Moved KFP backend visualization tests to GitHub Actions by @hbelmiro in #10984
• test: Migrate KFP SDK Upgrade Test to GHA by @DharmitD in #10989
• test: Migrate KFP SDK Runtime Code Test to GHA by @DharmitD in #10991
• chore(components): Bump image version for Structured Data pipelines by @copybara-service in #11000
• chore(components): Fix bug due to protobuf library being upgraded by pinning protobuf version by @copybara-service in #11007
• chore(components): Add target_field_name as input parameters to llm_evaluation_preprocessor component to support gemini model's input and output schema by @copybara-service in https://github.co...

KFP SDK v2.8.0

Release of the KFP SDK only.

To install the KFP SDK:

pip install kfp==2.8.0

For changelog, see release notes.

Version 2.2.0

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.7 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

What's Changed

• feat(components): Report TensorBoard metrics for preview.llm.rlhf_pipeline in real time by @copybara-service in #10595
• feat(kubernetes_platform): Update kubernetes_platform go package to include generic ephemerl volume by @abaland in #10602
• fix(metadata envoy): upgrade envoy and config from 1.12 to 1.27 by @freefood89 in #10589
• fix(components): Remove the unused resolve_data_paths from function_based by @copybara-service in #10606
• chore: Add Question issue template by @rimolive in #10557
• feat(kubernetes_platform): Update kubernetes_platform go package to include node affinities and pod (anti)affinities by @cjidboon94 in #10583
• chore(components): Add test machine spec support to preview.llm pipelines by @copybara-service in #10616
• fix(components): Ensure preview.llm.rlhf_pipeline runs if no tensorboard_id is provided by @copybara-service in #10626
• fix(components): Remove the unused functions from function_based by @copybara-service in #10613
• feat(components): Add model name preprocess component; Use publisher model if user uploaded model is non-tuned by @copybara-service in #10620
• feat(components): add task_type as a parameter to rlaif by @copybara-service in #10607
• feat(components): AutoSxS GA pending release by @copybara-service in #10628
• fix(components): Update service account comment by @copybara-service in #10636
• chore(components): GCPC 2.12.0 Release by @copybara-service in #10635
• fix(samples): Fix the loop_output.py example to handle the new parallel loop type requirement by @Tomcli in #10637
• feat(components): Added support for text-bison@002 to preview.llm.rlhf_pipeline by @copybara-service in #10641
• chore(components): GCPC 2.13.0 Release by @copybara-service in #10644
• fix(components): Make AutoSxS autorater_prompt_parameters required by @copybara-service in #10621
• docs(components): internal by @copybara-service in #10649
• chore(components): Update kserve component to v0.12.0 by @Tomcli in #10652
• fix(backend): Update backend common code and integration tests with updated API Service Params by @gmfrasca in #10640
• fix(Backend + SDK): Add missing optional field to SecretAsVolume and … by @revit13 in #10550
• chore(manifests): refactor manifests for kustomize5 compatibility. Part of #10053 by @rawc0der in #10087
• fix(components): Fix model eval import error in text generation/classification eval pipeline by @copybara-service in #10655
• chore(components): GCPC 2.13.1 Release by @copybara-service in #10666
• chore(deps): bump follow-redirects from 1.6.1 to 1.15.6 in /frontend by @dependabot in #10575
• chore(deps): bump pillow from 10.0.1 to 10.3.0 in /backend/src/apiserver/visualization by @dependabot in #10658
• fix(components): Remove the unused generate_default_instruction and resolve_upload_location from function_based by @copybara-service in #10638
• chore(deps): bump express from 4.18.2 to 4.19.2 in /frontend by @dependabot in #10639
• chore: Update argo images to 3.4.16 by @chensun in #10618
• feat(components): Use larger base reward model when tuning text and chat variants of bison@001 with the preview.llm.rlhf_pipeline by @copybara-service in #10663
• chore(deps): bump follow-redirects from 1.5.10 to 1.15.6 in /frontend/server by @dependabot in #10574
• chore(sample): add note about secret needing to be pre-created by @gregsheremeta in #10659
• Chore(components): Clean up old ibm components by @Tomcli in #10680
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10683
• feat(sdk+backend): Add support for generic ephemeral volume by @abaland in #10605
• feat(components): Use larger base reward model when tuning t5-xxl with the preview.llm.rlhf_pipeline by @copybara-service in #10665
• chore(backend): Promote @rimolive as the backend reviewer by @Tomcli in #10689
• chore(sdk): release kfp-kubernetes 1.2.0 by @connor-mccarthy in #10692
• chore(sdk): make kfp-kubernetes release instructions public by @connor-mccarthy in #10693
• chore(kfp-kubernetes): change type of affinity weight to int32 by @cjidboon94 in #10671
• fix(components): Fix image version parameter in rl pipelines by @copybara-service in #10698
• docs(components): internal by @copybara-service in #10694
• feat(backend): Merge kfp-tekton backend code by @rimolive in #10678
• feat(backend): Upgrade argo to v3.4.16 by @gmfrasca in #10568
• feat(backend): add namespace & prefix scoped credentials to kfp-launcher config for object store paths by @HumairAK in #10625
• chore(deps): bump sqlparse from 0.4.4 to 0.5.0 in /backend/src/apiserver/visualization by @dependabot in #10700
• feat(components): add resolve_reference_model_metadata to rlhf_preprocessor component by @copybara-service in #10612
• feat(components): Move AutoSxS pipeline to v1 directory by @copybara-service in #10701
• feat(components): add resolve_machine_spec and resolve_refined_image_uri to rlhf_preprocessor component by @copybara-service in #10608
• fix(components): remove default prediction column names in evaluation classification component to fix incorrect column names for bigquery data source by @copybara-service in #10708
• feat(components): Introduce placeholders: SERVICE_ACCOUNT_PLACEHOLDER, NETWORK_PLACEHOLDER, PERSISTENT_RESOURCE_ID_PLACEHOLDER and ENCYRPTION_SPEC_KMS_KEY_NAME_PLACEHOLDER. In addition, use PERSISTENT_RESOURCE_ID_PLACEHOLDER as the default value of persistent_resource_id for CustomTrainingJobOp and create_custom_training_job_op_from_component. With this change, custom job created without explicitly setting persistent_resource_id will inherit job level persistent_resource_id, if Persistent Resource is set as job level runtime by @copybara-service in #10650
• chore: Add kfp-tekton integration tests and manifests by @rimolive in #10702
• No public description by @copybara-service in #10726
• chore(components): Drop GCPC Python 3.7 by @copybara-service in #10730
• feat(components): use rlhf_preprocessor to replace the current value_exists call in rlhf by @copybara-service in #10584
• chore(components): Drop support for Python 3.7 in GCPC by @copybara-service in #10735
• feat(components): internal by @copybara-service in #10707
• feat(components): Expand regions supported by preview.llm.rlhf_pipeline by @copybara-service in #10710
• chore(components): Update AutoSxS and RLHF image tags by @copybara-service in #10749
• chore(components): Change the warning for Python 3.8 by @copybara-service in #10755
• feat(components): migrate function_based resolve_num_microbatches to rlhf_preprocessor component by @copybara-service in #10604
• chore(components): GCPC 2.14.0 Release by @copybara-service in #10754
• feat(components): migrate function_based convert_to_delimited_string to rlhf_preprocessor component by @copybara-service in https://github.com/kube...

Version 2.1.0

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.7 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log

What's Changed

• feat(components): Bump image tag used by preview.llm pipelines by @copybara-service in #10295
• feat(components): Add num_microbatches to _implementation.llm training components by @copybara-service in #10248
• chore(sdk): remove cleanup param in local init #localexecution by @connor-mccarthy in #10293
• feat(sdk): add local execution skeleton #localexecution by @connor-mccarthy in #10292
• fix(components): Use llama-2-7b for the base reward model when tuning llama-2-13 with the preview.llm.rlhf_pipeline by @copybara-service in #10249
• feat(sdk): add local task dispatcher validation and ExecutorInput construction #localexecution by @connor-mccarthy in #10298
• test(sdk): install kfp-pipeline-spec from source in kfp sdk tests by @connor-mccarthy in #10300
• fix(sdk): use kfp.dsl.types to replace kfp.components.types Fixes #10282 by @hsinhoyeh in #10283
• feat(components): Enable text generation pipeline to generate row based metrics by @copybara-service in #10296
• test(sdk): add placeholder_utils_test.py by @connor-mccarthy in #10301
• feat(component): Migrate AutoSxS pipeline to preview and move related files to _implementation/llm directory to help Model Eval team use side by side metrics as part of their pipeline by @copybara-service in #10294
• feat(components): Release new LLM Eval image version 0.5 by @copybara-service in #10313
• chore(components): release GCPC 2.7.0 by @copybara-service in #10303
• fix(components): Add autosxs_pipeline to the all variable for the preview/model_evaluation directory by @copybara-service in #10317
• chore(components): release GCPC 2.8.0 by @copybara-service in #10316
• feat(components): Add better docstrings for AutoSxS by @copybara-service in #10320
• test(components): INTERNAL by @copybara-service in #10314
• feat(sdk): add subprocess task handler #localexecution by @connor-mccarthy in #10302
• fix(components): Use large_model_reference as model_reference_name when uploading models from preview.llm.rlhf_pipeline instead of hardcoding value as text-bison@001 by @copybara-service in #10321
• test(sdk): add kfp-kubernetes execution tests by @connor-mccarthy in #10304
• fix(components): Resolve unique model display name on each preview.llm.rlhf_pipeline run instead of reusing cached result by @copybara-service in #10322
• fix(components): Upload the tuned adapter to Model Registry instead of model checkpoint from preview.llm.rlhf_pipeline by @copybara-service in #10323
• feat(sdk): add local execution output collection #localexecution by @connor-mccarthy in #10325
• chore(sdk): fix use of invalid escape sequence in tests by @connor-mccarthy in #10310
• feat(sdk): add local execution logging #localexecution by @connor-mccarthy in #10326
• chore(components): Fix argument description by @copybara-service in #10327
• feat(backend): preserve querystring in pipeline root (fixes #10318) by @TobiasGoerke in #10319
• feat(components): change output format to allow possible post eval by @copybara-service in #10281
• fix(frontend): Add disableParsingRawHTML option for markdown-to-jsx component by @zijianjoy in #10315
• feat(sdk): add DockerRunner #localexecution by @connor-mccarthy in #10328
• feat(sdk): add special dsl.OutputPath read logic #localexecution by @connor-mccarthy in #10334
• chore(sdk): write local execution logs to stdout #localexecution by @connor-mccarthy in #10330
• feat(sdk): support local Container Component execution #localexecution by @connor-mccarthy in #10333
• feat(components): Update RLHF env vars to handle empty string by @copybara-service in #10331
• feat(components): Output errors as a separate table from Arbiter by @copybara-service in #10329
• docs(components): Fix AutoSxS docstring formatting by @copybara-service in #10341
• chore(components): add json_escape placeholder util by @copybara-service in #10351
• fix(sdk): fix presentation of strings in local execution #localexecution by @connor-mccarthy in #10353
• fix(sdk): remove redundant newline character in local DockerRunner logs #localexecution by @connor-mccarthy in #10354
• test(sdk): improve KFP SDK local runner test safety #localexecution by @connor-mccarthy in #10336
• feat(backend): Enable logging for KFP components by @DharmitD in #10288
• feat(sdk): remove local execution feature flag #localexecution by @connor-mccarthy in #10355
• fix(sdk): permit empty local execution outputs #localexecution by @connor-mccarthy in #10338
• feat(sdk): support Concat and IfPresent placeholder in local container component execution #localexecution by @connor-mccarthy in #10348
• test(sdk): add test for local execution of None default parameter #localexecution by @connor-mccarthy in #10339
• chore(sdk): release KFP SDK 2.5.0 by @connor-mccarthy in #10364
• chore(sdk): depend on protobuf 4 in kfp-pipeline-spec by @connor-mccarthy in #10305
• docs(components): Document AutoML Tables util functions by @copybara-service in #10359
• feat(components): Implement the feature store grounding pipeline by @copybara-service in #10332
• feat(components): update eval pipeline documentation to clarify the required pipeline parameters by @copybara-service in #10366
• docs(components): update BigqueryQueryJobOp docs on container args limit by @copybara-service in #10380
• chore(components): GCPC 2.8.1 release by @copybara-service in #10381
• feat(kubernetes_platform): Update kubernetes_platform go package to include pod labels and annotations by @Tomcli in #10357
• chore(sdk): depend on protobuf 4 in kfp and kfp-kubernetes by @connor-mccarthy in #10307
• feat(components): Implement new output format of inference component by @copybara-service in #10375
• chore(components): rollback GCPC 2.8.1 release by @copybara-service in #10385
• chore(sdk): release KFP SDK 2.6.0 by @connor-mccarthy in #10386
• chore(components): Add v1.model.ModelGetOp components by @copybara-service in #10226
• chore(sdk): release kfp-kubernetes 1.1.0 by @connor-mccarthy in #10387
• chore(sdk): update kfp-kubernetes docs versions and release scripts by @connor-mccarthy in #10388
• docs(sdk): fix kfp-kubernetes docs build error by @connor-mccarthy in #10389
• chore(components): Sync AutoML components by @copybara-service in #10372
• fix(components): Update base image for KFP lightweight component for VPC SC compliance by @copybara-service in #10374
• chore: Fix metrics visualization v2 sample by @rimolive in #10399
• docs(sdk): add kfp.local to reference docs #localexecution by @connor-mccarthy in #10395
• chore(components): migrate GCPC to protobuf 4; require KFP>=2.6.0 by @copybara-service in #10401
• feat(components): Support scheduling and labels in utils.bui...

KFP SDK v2.7.0

Release of the KFP SDK only.

To install the KFP SDK:

pip install kfp==2.7.0

For changelog, see release notes.

KFP SDK v2.6.0

Release of the KFP SDK only.

To install the KFP SDK:

pip install kfp==2.6.0

For changelog, see release notes.

KFP SDK v2.5.0

Release of the KFP SDK only.

To install the KFP SDK:

pip install kfp==2.5.0

For changelog, see release notes.

Version 2.0.5

To deploy Kubeflow Pipelines in an existing cluster, follow the instruction in here or via UI here

Install python SDK (python 3.7 above) by running:

python3 -m pip install kfp kfp-server-api --upgrade

See the Change Log