Native compilation for openshift and knative consumes 4 times more RAM and takes 4 times longer

[fedinskiy]

Describe the bug

I have an application, which uses openshiftm knative and funqy. When I compile the application in native mode using Quarkus 3.5.2 it requires (via property quarkus.native.native-image-xmx) 4 gigabytes of RAM and the compilation takes around 2 minutes. When I compile the same application using Quarkus 3.14.3m, the compilation requires at least 16 gigabytes (it goes deadlocked at 8) and takes somewhere between eight and ten minutes.

Expected behavior

The perfomance should not degrade that much

Actual behavior

The resource consumption became much worse

How to Reproduce?

1. git clone https://github.com/fedinskiy/quarkus-test-suite -b reproducer/funqy-consumption
2. cd quarkus-test-suite/funqy/knative-events
3. mvn clean verify -P root-modules -Dnative -Dquarkus.platform.version=3.14.3 -Dquarkus.native.native-image-xmx=16g
   or
   mvn clean verify -P root-modules -Dnative -Dquarkus.platform.version=3.5.2 for comparision

Output of uname -a or ver

6.10.4-200.fc40.x86_64

Output of java -version

Java version: 21.0.1, vendor: Eclipse Adoptium

Mandrel or GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.14.3

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)

Additional information

The issue were previously reported as #37142 and #38683. Maybe one of these should be reopened, since the same problem affects 3.8.6 as well.

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @geoand (knative,openshift), @iocanel (knative,openshift), @matejvasek (funqy), @patriot1burke (funqy), @zakkak (mandrel,native-image)

[gsmet]

My wild guess is that it's because of the Kubernetes Client and not really related to Quarkus. Could you compare the size of the binaries?

Now I'm not entirely sure it will improve things but the first thing to do would be to depend on the quarkus-openshift-client extension instead of depending directly on the Fabric8 client.

[michalvavrik]

Sorry if I misunderstood your comment, but I'll try to react:

My wild guess is that it's because of the Kubernetes Client and not really related to Quarkus.

IIUC yes, exactly. However Quarkus is managing this client and using it in the extension we use. I didn't open this issue, but my understanding was that is the reason why it is opened here.

I think developers behind Fabric8 client are also integrating it into Quarkus K8/OpenShift together with @iocanel . @manusa right? So we can directly improve things by only bumping Fabric8 versions that doesn't cause this. Maybe we could have a test considering this is recurring issue?

Now I'm not entirely sure it will improve things but the first thing to do would be to depend on the quarkus-openshift-client extension instead of depending directly on the Fabric8 client.

quarkus-openshift-client rely on Fabric8 client, I glanced at it's deployment module and can't see there any buildtime magic except of substitutions (which itself surprise me) But let's wait for answer from experts.

[fedinskiy]

It seem that using quarkus-openshift-client instead of fabric8's knative-client and openshift-client indeed helps, and removes the need for the 16GBs compilation memory and lowers compilation time to 4-6 minutes. But as Michal mentioned earlier, it would be nice to know the reasons for this behavior

[michalvavrik]

I think that's fantastic and I'd really like to understand how that is possible just out of personal interest to understand things. Let's see if anyone can give me a hint. Thank you

[geoand]

cc @manusa @metacosm

[gsmet]

Part of the work of an extension is to optimize the behavior in native and that's what is done in these extensions.

If you have a look at KubernetesClientProcessor for instance, you will see IgnoreJsonDeserializeClassBuildItem being produced that will avoid registering some of the classes for reflection that would traditionally be registered for reflection by the Jackson extension, which could trigger loading a big part of the Kubernetes model that is massive.

IIRC this particular extension was highly problematic in the past and some work was done precisely to solve this issue. You can probably find some pointers by looking in the issues/PRs or by having a look at the history of the Kubernetes Client extension (and friends as there are also a few internal extensions).

[michalvavrik]

Understood, thank you. @fedinskiy is there anything else to this issue or are you going to close it?

[manusa]

@manusa right? So we can directly improve things by only bumping Fabric8 versions that doesn't cause this. Maybe we could have a test considering this is recurring issue?

The Fabric8 clients should be used from the Quarkus extensions for the reasons Guillaume pointed out, especially when performing a native image build.
Even if the application works when compiled to native, it's likely that it doesn't behave as expected.

For JVM mode, I don't think there should be a problem though.

[michalvavrik]

Alright, thank you for your time.