Please use jna.jar including libjnidispatch.so for android-x86

[PaulAtBanno]

problem

sbt-socket-server fails with an unresolved link exception on libjnidispatch.so when run under Termux on Android devices, such as Chromebooks.

expectation

Running sbt succeeds.

notes

The issue is sbt's use of JNA, and a default build of jna.jar. This .jar archive includes a reasonably comprehensive number of implementations of libjnidispatch, per-platform. However, android in general appears to be missing, and android-x86 in particular. Recent Chromebooks running Chrome OS also support running Android applications, but only i686 (32-bit) binaries. One such Android application is Termux, which provides a reasonably complete "Linux" userspace, including packages managed by the Debian package management system. The termux-packages project provides the patches and build infrastructure for Termux packages. One such package is openjdk-9-jre-headless, making it possible to run JDK 9 in Termux on Android devices. Including an android-x86 libjnidispatch.so in the jna.jar used by sbt would enable sbt to work properly in this JRE in Termux on (x86_64) Chromebooks. Including other Android CPU targets would also be desirable, of course.

sbt version: 1.1.0

[eed3si9n]

background

Here's from JNA's com.sun.jna.Native documentation:

When JNA classes are loaded, the native shared library (jnidispatch) is loaded as well. An attempt is made to load it from the any paths defined in jna.boot.library.path (if defined), then the system library path using System.loadLibrary(java.lang.String), unless jna.nosys=true. If not found, the appropriate library will be extracted from the class path (into a temporary directory if found within a jar file) and loaded from there, unless jna.noclasspath=true. If your system has additional security constraints regarding execution or load of files (SELinux, for example), you should probably install the native library in an accessible location and configure your system accordingly, rather than relying on JNA to extract the library from its own jar file.

To avoid the automatic unpacking (in situations where you want to force a failure if the JNA native library is not properly installed on the system), set the system property jna.nounpack=true.

So first it will look for jna.boot.library.path, so I think that's what you could pass in.

When jna.boot.library.path is not set, it says would've called System.loadLibrary(...), looking for things in java.library.path. However, we found this to be fragile/broken because java.library.path defaults to PATH on Windows. This means that if any previous version of jnidispatch were found, JNA will throw "There is an incompatible JNA native library installed on this system" error. (sbt/io#110). To workaround this, we programmatically set jna.nosys=true (#3833).

Finally, JNA will attempt to load jnidispatch from the classpath. This could be promising since build users can customize their global.sbt to append libraries as plugins.

solution ideas

• As noted above, could you try adding android-x86.jar from ~/.sbt/1.0/plugins/.
• When jna.boot.library.path is not set, maybe sbt could set it to somewhere relative to ~/.sbt/1.0/. This would mean that we'd have to stick to the compatible JNA version during the course of sbt 1.x. Otherwise change in build.properties could fail the build.
• Allow jna.sys=true that would opt-in System.loadLibrary(...). Then Chromebook users can put things into java.library.path. (JNA binary compatible problem will remain)
• If there's a more general upstream fix on JNA that might be better, so you could take up with them. FWIW https://github.com/java-native-access/jna/tree/4.5.0/dist shows android-x86 etc.

[cunei]

For the record, the android jnidispatch can be pulled in by using something like:

def aar(m: ModuleID): ModuleID =
  m.artifacts(Artifact(m.name, "aar", "aar")) exclude (
    "com.google.android", "support-v4") exclude (
    "com.google.android", "support-v13")

... += aar("net.java.dev.jna" % "jna" % "4.5.0")

[PaulAtBanno]

Thanks, Eugene! Some new results:

1. Placing the distributed android-x86.jar in ~/.sbt/1.0/plugins didn't work. This makes sense for a couple of reasons: first, because the "resource path" searched at the time the JNA bootstrap library is looked for is sbt's "boot classpath," i.e. before an sbt version is even established, and so presumably limited to whatever is associated with sbt-launcher rather than sbt proper, and secondly, because the contents of android-x86.jar is only META-INF and libjnidispatch.so, the latter not even in the com/sun/jna/android-x86 path required in order to be found by a "resource path" search. So I strongly suspect using this .jar would fail even if it were otherwise "in the right place," and in fact I have difficulty ascertaining any circumstances under which simply placing the .jar on the classpath could work.
2. Placing my built libjnidispatch.so in /usr/lib/jvm/openjdk-9/lib and setting jna.boot.library.path accordingly in /usr/conf/sbtopts "worked," in the sense that running sbt then gave me an error to the effect that the ELF library assumed the execstack ability, which the VM would try to work around, but a better solution would be to run execstack -c on the library, or build it without execstack in the first place. I couldn't find an execstack utility for Termux, so I went back to the source and added -z noexecstack to the CFLAGS. Running sbt with the resulting libjnidispatch.so library now gives me an ExecutionException: Boxed Error, which I strongly suspect stems from the JNA bootstrap library attempting to execute something on the stack since, after all, it's a stub library whose whole purpose is to trampoline to JNI!

So I'm now stuck. One thing that would be good to know is how to get a stacktrace for the [error] output from (I presume) sbt-launcher, because my understanding is that a Boxed Error wraps the underlying exception thrown in the context of a failed Future. That way I could confirm or refute my suspicion about what's going on now. However, I'm increasingly concerned that JNA expects to be able to execute code in contexts that are forbidden, either by the termux-chroot environment, the OpenJDK-9 JVM, Android, or the underlying Chrome OS environment.

Is there some way to run the sbt server without JNA, possibly at the cost of some lost functionality and/or responsiveness, e.g. watch-polling lag?

Thanks so much!

[PaulAtBanno]

Well, I don't know what I've been doing wrong, but it occurred to me that I could just extract libjnidispatch.so from the android-x86.jar from the 4.5.0 release (to match the version used by sbt), as opposed to building it from source. And when I put that .so in my /usr/lib/jvm/openjdk-9/lib and set jna.boot.library.path in sbtopts accordingly, the sbt-server starts up just fine in Termux on my Chromebook.

So that mystery is "solved," in the sense that all I care about is that it works (someone else can look into why building from source doesn't seem to—at least cross-compiling with the current Android NDK on macOS), but I'll leave this issue, which refers to the absence of Android support in sbt as distributed, open.