Blueprint proposals

[ReLoutre]

THIS ISSUE IS IN A WIP STATE.
Would like to propose a few blueprints that may help users understand what could be done with Kestra.
I would appreciate any help to improve them, feel free to propose, comment, test, etc

disc_space_hard_purges

id: disc_space_hard_purges
namespace: blueprints
description: "This flow may be triggered to clean the storage of this kestra instance on a selected period and while targeting specific item types."

labels:
  type: KESTRA_MANAGMENT_TOOL

inputs:
  - id: start
    type: DATETIME
    required: true

  - id: end
    type: DATETIME
    required: true

  - id: what_to_purge
    type: MULTISELECT
    values:
      - LOGS
      - EXECUTIONS
      - METRICS
      - STORAGE

  - id: allow_destructive_action
    type: STRING
    description: "BE CAREFUL !! HIGLY DESTRUCTIVE ACTION WILL BE PERFORMED, type 'ALLOW' if you're sure you want to destroy things you listed before."
    dependsOn:
      inputs:
        - start
        - end
        - what_to_purge
      condition: "{{ inputs.start is not empty and inputs.end is not empty and inputs.what_to_purge is not empty}}"
    required: true
    validator: ^ALLOW$

tasks:
  - id: logs_cleaning
    type: io.kestra.plugin.core.log.PurgeLogs
    startDate: "{{ inputs.start }}"
    endDate: "{{ inputs.end }}"
    runIf: "{{ inputs.what_to_purge contains 'LOGS' }}"

  - id: logs_bis_cleaning
    type: io.kestra.plugin.core.execution.PurgeExecutions
    purgeExecution: false
    purgeMetric: false
    purgeStorage: false
    purgeLog: true
    startDate: "{{ inputs.start }}"
    endDate: "{{ inputs.end }}"
    runIf: "{{ inputs.what_to_purge contains 'LOGS' }}"

  - id: storage_cleaning
    type: io.kestra.plugin.core.execution.PurgeExecutions
    purgeExecution: false
    purgeMetric: false
    purgeStorage: true
    purgeLog: false
    startDate: "{{ inputs.start }}"
    endDate: "{{ inputs.end }}"
    runIf: "{{ inputs.what_to_purge contains 'STORAGE' }}"

  - id: metrics_cleaning
    type: io.kestra.plugin.core.execution.PurgeExecutions
    purgeExecution: false
    purgeMetric: true
    purgeStorage: false
    purgeLog: false
    startDate: "{{ inputs.start }}"
    endDate: "{{ inputs.end }}"
    runIf: "{{ inputs.what_to_purge contains 'METRICS' }}"

  - id: execs_cleaning
    type: io.kestra.plugin.core.execution.PurgeExecutions
    purgeExecution: true
    purgeMetric: false
    purgeStorage: false
    purgeLog: false
    startDate: "{{ inputs.start }}"
    endDate: "{{ inputs.end }}"
    runIf: "{{ inputs.what_to_purge contains 'EXECUTIONS' }}"

get_all_namespaces_via_api_req

id: get_all_namespaces
namespace: blueprints

variables:
  kestra_local_uri: "http://localhost:8080/kestra"

labels:
  type: TOOL

tasks:
  - id: get_namespaces_nbr
    type: io.kestra.plugin.core.http.Request
    method: GET
    uri: "{{ vars.kestra_local_uri }}/api/v1/namespaces/search?size=1"
    headers:
      Authorization: "{{ secret(KESTRA_INTERNAL_API_TOKEN) }}"

  - id: get_all_namespaces
    type: io.kestra.plugin.core.http.Request
    method: GET
    uri: "{{ vars.kestra_local_uri }}/api/v1/namespaces/search?size={{ json(outputs.get_namespaces_nbr['body']).total }}"
    headers:
      Authorization: "{{ secret(KESTRA_INTERNAL_API_TOKEN) }}"

outputs:
  - id: namespaces
    type: ARRAY
    value: |
      [{% for elem in json(outputs.get_all_namespaces["body"]).results %}"{{ elem.id }}"{% if not loop.last %},{% endif %}{% endfor %}]

  - id: total
    type: INT
    value: "{{ json(outputs.get_namespaces_nbr['body']).total }}"

save_kestra_to_git

I used gitlab for this one, it needs test using github API.

id: save_kestra_to_git
description: "This flow intend to save weekly to your git repository all of your namespaces : flows, files and KVs. Information on a release branch_target may be inputed to automate its deployment."
namespace: blueprints

variables:
    your_git_repository: https://gitlab.com/your_orga/some_kestra_repo
    kestra_local_uri: http://localhost:8080
    your_project_id: "254"

inputs:
  - id: make_new_release
    type: BOOLEAN
    defaults: false

  - id: tag_name
    type: STRING
    dependsOn:
        inputs:
         - make_new_release
        condition: "{{ inputs.make_new_release }}"
    required: true

  - id: release_name
    type: STRING
    dependsOn:
        inputs:
         - make_new_release
        condition: "{{ inputs.make_new_release }}"
    required: true
    defaults: "Latest"
    description: "If set to 'Latest' the flow will create a pull request from this latest state to the MAIN branch from which saves are pulled by default, aka : THE latest release."

  - id: additional_description
    type: STRING
    dependsOn:
        inputs:
         - make_new_release
        condition: "{{ inputs.make_new_release }}"
    required: false

labels:
  type: KESTRA_MANAGMENT_TOOL

tasks:
  - id: get_namespaces
    type: io.kestra.plugin.core.flow.Subflow
    flowId: get_all_namespaces
    namespace: blueprints

  - id: for_each_namespace
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ outputs.get_namespaces['outputs']['namespaces'] }}"
    concurrencyLimit: 1
    tasks:
      - id: pushes_and_kv_storing
        type: io.kestra.plugin.core.flow.Parallel
        tasks:
          - id: kv_storing
            type: io.kestra.plugin.core.flow.Sequential
            tasks:
              - id: get_keys
                type: io.kestra.plugin.core.http.Request
                method: GET
                uri: "{{ vars.kestra_local_uri }}/api/v1/namespaces/{{ parent.taskrun.value }}/kv/"
                headers:
                  Authorization: "{{ secret(KESTRA_INTERNAL_API_TOKEN) }}"
              - id: for_each_key
                type: io.kestra.plugin.core.flow.ForEach
                concurrencyLimit: 0
                values: "{{ outputs.get_keys[parent.taskrun.value].body }}"
                tasks:
                  - id: retrieve_values
                    type: io.kestra.plugin.core.http.Request
                    method: GET
                    uri: "{{ vars.kestra_local_uri }}/api/v1/namespaces/{{ parents[0].taskrun.value }}/kv/{{ json(taskrun.value).key }}"
                    headers:
                      Authorization: "{{ secret(KESTRA_INTERNAL_API_TOKEN) }}"
          - id: pushes
            type: io.kestra.plugin.core.flow.Sequential
            tasks:
              - id: push_files
                type: io.kestra.plugin.git.PushNamespaceFiles
                commitMessage: Add files from `{{ parents[0].taskrun.value }}` namespace.
                gitDirectory: "_files/{{ parents[0].taskrun.value }}"
                namespace: "{{ parents[0].taskrun.value }}"
                url: "{{ vars.your_git_repository }}"
                password: "{{ secret(KESTRA_GITLAB_API_TOKEN) }}"
                username: "Depends on if you use a token or not."
                dryRun: false
              - id: push_flows
                type: io.kestra.plugin.git.PushFlows
                gitDirectory: "_flows/{{ parents[0].taskrun.value }}"
                commitMessage: Add flows from `{{ parents[0].taskrun.value }}` namespace.
                includeChildNamespaces: false
                sourceNamespace: "{{ parents[0].taskrun.value }}"
                targetNamespace: "{{ parents[0].taskrun.value }}"
                url: "{{ vars.your_git_repository }}"
                password: "{{ secret(KESTRA_GITLAB_API_TOKEN) }}"
                username: "Depends on if you use a token or not."

  - id: set_wd
    type: io.kestra.plugin.core.flow.WorkingDirectory
    inputFiles:
      kvStore.json: |
        {% for elem in outputs.retrieve_values %}{"namespace":"{{ elem.key }}","keys":[{% for ele in (elem.value) %}{"key":"{{ json(ele.key).key }}","body":{{ ele.value.body }}}{% if not loop.last %},{% endif %}{% endfor %}]}
        {% endfor %}
    tasks:
      - id: storeKv
        type: io.kestra.plugin.core.namespace.UploadFiles
        conflict: OVERWRITE
        files:
          - kvStore.json
        namespace: kestra_export_import
        destination: "/_kv/"

  - id: push_kv_stores
    type: io.kestra.plugin.git.PushNamespaceFiles
    commitMessage: Add KV stores.
    gitDirectory: "."
    files: /_kv/**
    url: "{{ vars.your_git_repository }}"
    password: "{{ secret(KESTRA_GITLAB_API_TOKEN) }}"
    username: "Depends on if you use a token or not."

  - id: deleteStoredKv
    type: io.kestra.plugin.core.namespace.DeleteFiles
    namespace: kestra_export_import
    files: /_kv/**

  - id: make_release
    type: io.kestra.plugin.core.http.Request
    runIf: "{{ inputs.make_new_release }}"
    method: POST
    contentType: application/json
    uri: "https://gitlab.com/api/v4/projects/{{ vars.your_project_id }}/releases?ref=kestra&access_token={{ secret(KESTRA_GITLAB_API_TOKEN) }}"
    body: |
      { "name": "{{ inputs.release_name }}", "tag_name": "{{ inputs.tag_name }}", "description": "{{ inputs.additional_description }}"}

  - id: create_pull_request
    type: io.kestra.plugin.core.http.Request
    runIf: "{{ inputs.release_name equals 'Latest' }}"
    method: POST
    contentType: application/json
    uri: "https://gitlab.com/api/v4/projects/{{ vars.your_project_id }}/merge_requests?title=lts_up&source_branch=kestra&target_branch=main&access_token={{ secret(KESTRA_GITLAB_API_TOKEN) }}"

triggers:
  - id: weekly_saves
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 0 7,14,21,28 * *"

Here's a example of a stored kvStore made by this flow :

{"namespace":"some_namesapce.psi.error_handling","keys":[{"key":"smtp_logs","body":{"type":"JSON","value":{"SERVER":"smtp1.your_orga.fr"}}}]}
{"namespace":"some_namesapce.kestra_export_import","keys":[{"key":"kestra_git_user","body":{"type":"STRING","value":"svc_kestra"}},{"key":"kestra_local_uri","body":{"type":"STRING","value":"http://localhost:8080/kestra"}}]}
{"namespace":"some_namesapce.nested_one.re_nested_one","keys":[{"key":"mq_serv","body":{"type":"STRING","value":"172.16.17.8:18083,172.16.17.8:18087,172.16.17.8:18091"}},{"key":"ldap_logs","body":{"type":"JSON","value":{"DN_ONE":"uid=svc_kestra,ou=bots,dc=one,dc=your_orga,dc=fr","DN_TWO":"uid=svc_kestra,ou=bots,dc=two,dc=your_orga,dc=fr","HOST_PERS":"172.16.9.8","HOST_ETU":"172.16.9.9","PORT":"389"}}},{"key":"win_rpc_vals","body":{"type":"JSON","value":{"SERVER":"10.1.1.98","USER":"BLOUP"}}}]}

And here's the expected layout of your save :

load_kestra_from_git

Works with a gitlab save made using the previous blueprint.

id: load_kestra_from_git
namespace: blueprints
description: "May be used to dowload and setup a new instance of Kestra (which may overlap with a currently deployed one) from a gitlab save made using the previous blueprint."

variables:
    your_git_repository: https://gitlab.com/your_orga/some_kestra_repo
    kestra_local_uri: http://localhost:8080
    your_project_id: "254"

labels:
  type: KESTRA_MANAGMENT_TOOL

inputs:
  - id: branch_target
    type: SELECT
    allowCustomValue: false
    values:
      - main
      - kestra
    required: true

tasks:
  - id: get_flows_targets
    type: io.kestra.plugin.core.http.Request
    method: GET
    uri: "https://gitlab.com/api/v4/projects/{{ vars.your_project_id }}/repository/tree/?ref={{ inputs.branch_target }}&recursive=false&path=_flows&access_token={{ secret(KESTRA_GITLAB_API_TOKEN) }}"

  - id: for_each_flows_folder
    type: io.kestra.plugin.core.flow.ForEach
    values: |
      [{% for elem in json(outputs.get_flows_targets['body']) %}"{{ elem.name }}"{% if not loop.last %},{% endif %}{% endfor %}]
    tasks:
      - id: pull_flows
        type: io.kestra.plugin.git.SyncFlows
        branch: "{{ inputs.branch_target }}"
        gitDirectory: "_flows/{{ taskrun.value }}"
        targetNamespace: "{{ taskrun.value }}"
        url: "{{ vars.your_git_repository }}"
        password: "{{ secret(KESTRA_GITLAB_API_TOKEN) }}"
        username: "Depends on if you use a token or not."
        dryRun: false

  - id: get_files_targets
    type: io.kestra.plugin.core.http.Request
    method: GET
    uri: "https://gitlab.com/api/v4/projects/{{ vars.your_project_id }}/repository/tree/?ref={{ inputs.branch_target }}&recursive=false&path=_files&access_token={{ secret(KESTRA_GITLAB_API_TOKEN) }}"

  - id: for_each_files_folder
    type: io.kestra.plugin.core.flow.ForEach
    values: |
      [{% for elem in json(outputs.get_files_targets['body']) %}"{{ elem.name }}"{% if not loop.last %},{% endif %}{% endfor %}]
    tasks:
      - id: pull_files
        type: io.kestra.plugin.git.SyncNamespaceFiles
        branch: "{{ inputs.branch_target }}"
        gitDirectory: "_files/{{ taskrun.value }}"
        namespace: "{{ taskrun.value }}"
        url: "{{ vars.your_git_repository }}"
        password: "{{ secret(KESTRA_GITLAB_API_TOKEN) }}"
        username: "Depends on if you use a token or not."
        dryRun: false

  - id: get_stored_kvs
    type: io.kestra.plugin.core.http.Request
    method: GET
    uri: "https://gitlab.com/api/v4/projects/{{ vars.your_project_id }}/repository/files/_kv%2FkvStore.json?ref={{ inputs.branch_target }}&recursive=false&path=_flows&access_token={{ secret(KESTRA_GITLAB_API_TOKEN) }}"

  - id: for_each_namespace_store
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ json(outputs.get_stored_kvs['body']).content | base64decode | split('\n') }}"
    tasks:
      - id: for_each_key
        type: io.kestra.plugin.core.flow.ForEach
        values: "{{ json(taskrun.value).keys }}"
        tasks:
          - id: put_key
            type: io.kestra.plugin.core.http.Request
            method: PUT
            contentType: application/json
            uri: "{{ kv('kestra_local_uri') }}/api/v1/namespaces/{{ json(parent.taskrun.value).namespace }}/kv/{{ json(taskrun.value).key }}"
            headers:
              Authorization: "{{ secret(KESTRA_INTERNAL_API_TOKEN) }}"
            body: |
              {% if json(taskrun.value).body.type equals "STRING" %}"{% endif %}{{ json(taskrun.value).body.value }}{% if json(taskrun.value).body.type equals "STRING" %}"{% endif %}

[MilosPaunovic]

This issue might need transferring to our blueprints repository, @Ben8t?

[Ben8t]

Yep, done ;)

Also @ReLoutre do not hesitate to submit PR directly so we can review and merge ;) Thanks for your contribution here !

[ReLoutre]

Hi @Ben8t,
Didn't do a PR for two reason :

• First and foremost, i didn't know where to put a blueprint PR nor do i know what format it should follow (I didn't even know there was a blueprint repo @MilosPaunovic XD).
• Second one, i was eager to have some review on those blueprints (mainly from the community), without external tests it doesn't seems good to provide it to the "official" solution.

Could be good to add a bit of doc (on the blueprints page i guess) to know where to post blueprints proposal (searched for it but didn't find). Or make a button (as for 'create an issue') which would link to a github template on the right repo.

Edit : I'lll make an issue for those feature requests.

[Ben8t]

Second one, i was eager to have some review on those blueprints (mainly from the community), without external tests it doesn't seems good to provide it to the "official" solution.

PR review is here for that 😉

Didn't get a change to review them fully, but at first glance there are probably too many tasks or properties in the disc_space_hard_purges, save_kestra_to_git and load_kestra_from_git. Worth adding pluginDefaults and simplify some git references (don't hesitate to hardcode some values instead of templating things with inputs: some users who begin with Kestra could be overloaded with too much information)

[ReLoutre]

Worth adding pluginDefaults and simplify some git references

Good point, I'll address that. Templated tasks could also help in some cases.

some users who begin with Kestra could be overloaded with too much information

While Kestra focuses heavily on onboarding newcomers, this approach can leave users stuck in a “beginner” phase. When I was starting out, I found it challenging to move beyond that stage. Advanced examples or blueprints would have been invaluable to me at that point.

The blueprints I’ve created aim to showcase real-world use cases and demonstrate how to effectively leverage Kestra’s tools. My goal is to make these flows more “end-user” friendly—whether for someone clicking "execute" or for flows called as subflows. I want to provide examples of “completed” flows—solid, well-designed ones—not just starter templates.

Edit : could add similar annotations for blueprints :

For these blueprints to be effective, they should be ready to use almost immediately after being copied. This brings me to the following point:

hardcode some values instead of templating things with inputs

Exactly what type of practice i would like to get rid of in some real life examples (not all of them but having a couple of it could be good).

I also feel that Kestra’s current documentation (e.g., YouTube videos, guides, blueprints) could benefit from more advanced examples to help users grow their skills. It’s great to attract new community members, but retaining and supporting their development is equally critical.

YET, I do know those blueprints lacks here and there, which is why i didn't made a PR right away and am glad to exchange with you on it.