Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the possibility to outputs sqllite or DuckDB file database #480

Open
Ben8t opened this issue Jan 13, 2025 · 1 comment
Open

Add the possibility to outputs sqllite or DuckDB file database #480

Ben8t opened this issue Jan 13, 2025 · 1 comment
Assignees
Labels
area/plugin Plugin-related issue or feature request enhancement New feature or request

Comments

@Ben8t
Copy link
Member

Ben8t commented Jan 13, 2025

Feature description

Using in-memory database system such SQLlite or DuckDB, user might want to pass the file database artifact from task to task. Currently it's possible to input a database:

id: ingest_database
namespace: kestra.qa

tasks:

    - id: create_db
      type: io.kestra.plugin.scripts.shell.Commands
      commands:
        - touch file.db
      outputFiles:
        - "file.db"

    - id: update
      type: io.kestra.plugin.jdbc.sqlite.Queries
      url: jdbc:sqlite:file.db
      sqliteFile: '{{ outputs.create_db.outputFiles["file.db"]}}'
      sql: |
        CREATE TABLE IF NOT EXISTS {{flow.id}}_{{ execution.id }} (
          "time" VARCHAR,
          "temperature_2m" VARCHAR,
          "latitude" VARCHAR,
          "longitude" VARCHAR,
          "city" VARCHAR
        );

        CREATE TABLE IF NOT EXISTS temperature (
          "time" VARCHAR,
          "temperature_2m" VARCHAR,
          "latitude" VARCHAR,
          "longitude" VARCHAR,
          "city" VARCHAR,
          primary key(time, city)
        );
      fetchType: STORE

But it's not possible to output the file.db and use it in downstream tasks

@Ben8t Ben8t added enhancement New feature or request area/plugin Plugin-related issue or feature request labels Jan 13, 2025
@Ben8t Ben8t added this to Issues Jan 13, 2025
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jan 13, 2025
@Ben8t
Copy link
Member Author

Ben8t commented Jan 13, 2025

Even using a simple

    - id: update
      type: io.kestra.plugin.jdbc.sqlite.Queries
      url: jdbc:sqlite:file.db
      sql: |
        CREATE TABLE IF NOT EXISTS {{flow.id}}_{{ execution.id }} (
          "time" VARCHAR,
          "temperature_2m" VARCHAR,
          "latitude" VARCHAR,
          "longitude" VARCHAR,
          "city" VARCHAR
        );

        CREATE TABLE IF NOT EXISTS temperature (
          "time" VARCHAR,
          "temperature_2m" VARCHAR,
          "latitude" VARCHAR,
          "longitude" VARCHAR,
          "city" VARCHAR,
          primary key(time, city)
        );
      fetchType: STORE

It's not possible to access the file.db

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/plugin Plugin-related issue or feature request enhancement New feature or request
Projects
Status: Backlog
Development

No branches or pull requests

2 participants