How to set a higher JVM heap size or spark driver/executor memory

[raksha1]

Is there a way to increase java heap size or spark driver/executor memory? The default JVM heap size of 4096MB is not enough and we are looking to increase the memory to 10GB at least.

How can we set a higher JVM heap size or spark memory?

We are running this job as a kubernetes cronjob and setting JAVA_OPTS environment variable (JAVA_OPTS="-Xms10g -Xmx10g") doesn't work.

Both memory request and limit are set to 16GB

[pavolloffay]

The heap size should be configurable via JAVA_OPTS https://github.com/jaegertracing/spark-dependencies#quick-start

[pavolloffay]

Could you please paste here pod spec?

[raksha1]

Here's the pod spec, started by the cronjob. I've removed irrelevant fields and changed names

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: jaeger
    app.kubernetes.io/component: spark-dependencies
  name: jaeger-spark-dependencies
  namespace: jaeger
spec:
  containers:
  - env:
    - name: STORAGE
      value: elasticsearch
    - name: ES_NODES
      value: <elasticsearch endpoint>
    - name: ES_NODES_WAN_ONLY
      value: "true"
    - name: JAVA_OPTS
      value: -Xms10g -Xmx10g
    image: jaegertracing/spark-dependencies:latest
    imagePullPolicy: Always
    name: jaeger-spark-dependencies
    resources:
      limits:
        cpu: "2"
        memory: 16Gi
      requests:
        cpu: 300m
        memory: 16Gi

I did exec into the pod and checked JAVA_OPTS env. It is set

JAVA_OPTS=-Xms10g -Xmx10g