Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: stars blueprint and namespace names #8

Merged
merged 1 commit into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 0 additions & 93 deletions dwh-and-analytics-1.yaml

This file was deleted.

91 changes: 59 additions & 32 deletions dwh-and-analytics.yaml
Original file line number Diff line number Diff line change
@@ -1,28 +1,24 @@
id: dwh-and-analytics
namespace: company.team
description: >
## Data Platform

Clone a [Git repository](https://github.com/kestra-io/dbt-example) and build
dbt models
namespace: tutorial
description: Data Warehouse and Analytics
tasks:
- id: dbt
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: clone_repository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-example
branch: master
url: https://github.com/kestra-io/dbt-demo
branch: main
- id: dbt_build
type: io.kestra.plugin.dbt.cli.DbtCLI
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
commands:
- dbt deps --project-dir dbt
- dbt build --project-dir dbt
- dbt deps
- dbt build
profiles: |
my_dbt_project:
jaffle_shop:
outputs:
dev:
type: duckdb
Expand All @@ -32,35 +28,66 @@ tasks:
fixed_retries: 1
threads: 16
timeout_seconds: 300
target: dev
extend:
title: Git workflow for dbt — pull latest dbt changes from GitHub and build dbt
models from your dbt project directory
description: >
This flow does the following:
target: dev
- id: python
type: io.kestra.plugin.scripts.python.Script
outputFiles:
- "*.csv"
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/duckdb:latest
script: >
import duckdb

1. Clones the latest dbt code from
[GitHub](https://github.com/kestra-io/dbt-example/tree/master)
import pandas as pd

2. Pulls a public [container image with the latest package
dependencies](https://github.com/kestra-io/examples/pkgs/container/dbt-duckdb)

3. Runs dbt CLI commands in a Docker container.
conn = duckdb.connect(database='dbt.duckdb', read_only=False)


Note how the dbt CLI commands use the `--project-dir dbt` flag. This is
important because the [example GitHub project we
use](https://github.com/kestra-io/dbt-example/tree/master) has the dbt
project code located in a dedicated folder.
tables_query = "SELECT table_name FROM information_schema.tables WHERE
table_schema = 'main';"

tables = conn.execute(tables_query).fetchall()

If your dbt project is located outside of the root directory in your Git
repository structure, make sure to use the `--project-dir dbt` flag.

# Export each table to CSV, excluding tables that start with 'raw' or
'stg'

for table_name in tables:
table_name = table_name[0]
# Skip tables with names starting with 'raw' or 'stg'
if not table_name.startswith('raw') and not table_name.startswith('stg'):
query = f"SELECT * FROM {table_name}"
df = conn.execute(query).fetchdf()
df.to_csv(f"{table_name}.csv", index=False)

conn.close()
extend:
title: Getting started with Kestra — a Data Warehouse and Analytics workflow example
description: >-
This flow is a simple example of a data warehouse and analytics use case. It
clones a dbt repository, builds the dbt project, and exports the data to CSV
files.


The flow has three tasks:

1. The first task clones a dbt repository.

2. The second task builds the dbt models and tests using DuckDB.

3. The third task exports the transformed data to CSV files.
tags:
- Git
- Getting Started
- Data
- dbt
- DuckDB
ee: false
- Git
- Python
- Docker
ee: true
demo: true
meta_description: This flow clones a dbt Git repository from GitHub and then
runs dbt CLI commands in a Docker container.
meta_description: This flow is a simple example of a data warehouse and
analytics use case. It clones a dbt repository, builds the dbt project, and
exports the data to CSV files.
6 changes: 3 additions & 3 deletions failure-alert-sentry.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
id: failure-alert-sentry
namespace: prod.monitoring
namespace: system
tasks:
- id: send_alert
type: io.kestra.plugin.notifications.sentry.SentryExecution
Expand All @@ -16,10 +16,10 @@ triggers:
- FAILED
- WARNING
- type: io.kestra.plugin.core.condition.ExecutionNamespaceCondition
namespace: prod
namespace: company
prefix: true
extend:
title: Send an alert to Sentry when any flow fails in the production namespace
title: Send an alert to Sentry when any flow fails in the company namespace
description: >-
This flow shows how to send an alert to Sentry when a flow fails. The only
required input is a DSN string value, which you can find when you go to your
Expand Down
3 changes: 1 addition & 2 deletions notify-about-github-stars-via-slack.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
id: notify-about-github-stars-via-slack
namespace: blueprints
namespace: company.team
inputs:
- id: repo
type: STRING
Expand Down Expand Up @@ -27,7 +27,6 @@ extend:

tags:
- API
- Getting Started
- Git
- Outputs
ee: false
Expand Down
2 changes: 1 addition & 1 deletion produce-to-rabbitmq.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ extend:

id: read_orders

namespace: blueprint
namespace: company.team


tasks:
Expand Down
2 changes: 1 addition & 1 deletion send-email-with-attachment.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
id: send-email-with-attachment
namespace: blueprint
namespace: company.team
tasks:
- id: dataset1
type: io.kestra.plugin.core.http.Download
Expand Down
3 changes: 1 addition & 2 deletions weaviate-load-and-query.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,7 @@ extend:
The last task performing a [Generative
Search](https://weaviate.io/developers/weaviate/starter-guides/generative#what-is-generative-search)
is currently disabled, as it requires an OpenAI API key and following the
[Weaviate getting started
documentation](https://weaviate.io/developers/weaviate/quickstart). You can
[Weaviate documentation](https://weaviate.io/developers/weaviate/quickstart). You can
enable it by removing the `disabled: true` line.
tags:
- AI
Expand Down
6 changes: 3 additions & 3 deletions zenduty-failure-alert.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
id: zenduty-failure-alert
namespace: prod.monitoring
namespace: system
tasks:
- id: send_alert
type: io.kestra.plugin.notifications.zenduty.ZendutyExecution
Expand All @@ -17,10 +17,10 @@ triggers:
- FAILED
- WARNING
- type: io.kestra.plugin.core.condition.ExecutionNamespaceCondition
namespace: prod
namespace: company
prefix: true
extend:
title: Send an alert to Zenduty when any flow fails in the production namespace
title: Send an alert to Zenduty when any flow fails in the company namespace
description: >-
This flow sends an alert to Zenduty when a production flow fails. The only
required input is an integration key string value. Check the [Zenduty
Expand Down