Enable agent instrumentation in native image by attaching agent at build time

[ziyilin]

Agent instrumentation is a runtime dynamic behavior which is not supported by GraalVM native image at present, but is an essential feature used in practice for real time monitoring, function enhancement without modifying original code and etc. Therefore, it's important to support agent instrumentation in native image.

We have discussed what kind of agent usage in JVM is possible to get supported in native image, and given an implementation in this PR. It handles agent class transformation in the similar ways as other Java dynamic features such as reflection and predefined classes: firstly records the class transformation events and dump the transformed class out in training runs; secondly loads them at build time to replace the original classes. JDK classes are manually transformed with GraalVM's class substitution mechanism(i.e. @TargetClass APIs) to avoid conflicts and disturbing the build process. This solution can support practical Java agent such as OTel in production environments.

However, as the training run cannot cover all possible paths, this solution may miss class transformations. An alternative improvement is to attach the agent at build time. GraalVM loads all classes in the classpath at build time, so all class transformations will be done by agent and seen by the compiler.

The most significant technique challenge is how to prevent JDK class instrumentation interference. Besides application class, the agent attached at build time can transform JDK classes which serve the GraalVM build framework as well. It means the building behaviors will be altered by the agent, but we can't even evaluate the impact because agent can do anything in the enhanced code. In a word, we want the agent to transform the JDK classes compiled into the native image, but don't want it to transform the JDK classes used by GraalVM framework. How could this be done?

One possible way is to attach another JVMTI agent to inspect the transformation at build time. The JVMTI agent can intercept the transformation of JDK classes, but only allow the transformation of application class. At this point, the next problem is how to handle JDK transformations. An easier way is let user manually transform with APIs provides in this PR. Another more automatic way is to shade the transformed JDK class to a different class, and use it to substitute the original one through new subclass of SubstitutionProcessor prior to AnnotationSubstitutionProcessor.

This approach requires to attach a JVMTI agent at build time. It may slow down the build process. It is a big downside of this solution. Or maybe there are other ways?