Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IonToJson requires x8-x10 more RAM then inital file size #164

Open
NovytskyiD opened this issue Jan 30, 2025 · 0 comments
Open

IonToJson requires x8-x10 more RAM then inital file size #164

NovytskyiD opened this issue Jan 30, 2025 · 0 comments
Labels
area/plugin Plugin-related issue or feature request bug Something isn't working

Comments

@NovytskyiD
Copy link

Describe the issue

On my VM with 8 CPU and 8 GB RAM io.kestra.plugin.serdes.json.IonToJson can convert a 600 Mb file in 2 minutes, but a 1 Gb file fails with a Java heap space error after 40 minutes. During this period, UI almost doesn't work and new flows don't start by schedule.

I use - JAVA_OPTS=-Xms4G -Xmx6G config.

flow that I run

- id: getParams
    type: io.kestra.plugin.jdbc.sqlserver.Queries
    url: jdbc:sqlserver://;trustServerCertificate=true
    username: 
    password: 
    sql: |
      DECLARE @DateFrom DATE = '2024-12-01'
        , @DateTo DATE = GETDATE()-1
      EXEC [DataBase].[dbo].[sp_Test] @DateFrom = @DateFrom, @DateTo = @DateTo;
    fetchType: STORE
  - id: resultToJson
    type: io.kestra.plugin.serdes.json.IonToJson
    from: "{{ outputs.getParams.outputs[0].uri }}"
    newLine: false

slack thread with discussion - https://kestra-io.slack.com/archives/C03FQKXRK3K/p1738239094441759

Environment

  • Kestra Version: 0.20.13
@NovytskyiD NovytskyiD added area/plugin Plugin-related issue or feature request bug Something isn't working labels Jan 30, 2025
@kestrabot kestrabot bot added this to Issues Jan 30, 2025
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/plugin Plugin-related issue or feature request bug Something isn't working
Projects
Status: Backlog
Development

No branches or pull requests

1 participant