Skip to content

Commit

Permalink
feat: deploy pipelines in batch from directory rather than from file
Browse files Browse the repository at this point in the history
  • Loading branch information
luca-serra committed Sep 26, 2023
1 parent b1224da commit e1a0486
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 14 deletions.
8 changes: 3 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,10 @@ gcloud services enable \
--project=$GCP_PROJECT_ID
```

4. Deploy the scheduled pipeline and its infrastructure:
4. Create a directory with the compiled pipeline(s) inside & deploy the scheduled pipeline(s) and its (their) infrastructure:

```bash
make deploy_scheduled_pipeline <path_to_local_pipeline_yaml_file>
make deploy_scheduled_pipeline <path_to_pipeline_templates_directory>
```

This command will:
Expand All @@ -96,11 +96,9 @@ This command will:
- Give the appropriate permissions to the service accounts.
- Upload the pipeline templates to the Artifact Registry repository.

Run this command as many times as you want to upload different pipelines.

> Note: you can use the dummy pipeline to test the scheduling:
```bash
make deploy_scheduled_pipeline examples/hello_world_pipeline.yaml
make deploy_scheduled_pipeline examples
```

## Sanity check
Expand Down
25 changes: 16 additions & 9 deletions bin/upload_pipeline_template.sh
Original file line number Diff line number Diff line change
@@ -1,17 +1,24 @@
#!/bin/bash -e

path_to_pipeline_template=$1
path_to_pipeline_templates_directory=$1
path_to_pipeline_templates_directory="${path_to_pipeline_templates_directory%/}"

region=$(grep 'region' scheduled_pipelines_config.yaml | sed -n 's/.*region:[^a-zA-Z0-9-]*\([a-zA-Z0-9-]*\).*/\1/p')
project_id=$(grep 'id' scheduled_pipelines_config.yaml | sed -n 's/.*id:[^a-zA-Z0-9-]*\([a-zA-Z0-9-]*\).*/\1/p')
repository_name=$(grep 'repository_name' scheduled_pipelines_config.yaml | sed -n 's/.*repository_name:[^a-zA-Z0-9-]*\([a-zA-Z0-9-]*\).*/\1/p')

echo "Uploading pipeline template to Artifact Registry (${region}-kfp.pkg.dev/${project_id}/${repository_name})..."
echo "Artifact Registry: ${region}-kfp.pkg.dev/${project_id}/${repository_name}"

curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-F tags=latest \
-F content=@${path_to_pipeline_template} \
https://${region}-kfp.pkg.dev/${project_id}/${repository_name}

echo "Pipeline template ${path_to_pipeline_template} successfully uploaded to Artifact Registry!"
for template_file in $path_to_pipeline_templates_directory/*.yaml; do
if [ ! -f "$template_file" ]; then
echo "No pipeline template found in $path_to_pipeline_templates_directory"
exit 1
else
echo "Uploading pipeline template $template_file..."
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-F tags=latest \
-F content=@${template_file} \
https://${region}-kfp.pkg.dev/${project_id}/${repository_name}
fi
done

0 comments on commit e1a0486

Please sign in to comment.