Systemless Support

[topjohnwu]

Hello @rovo89
Here I pull request to support systemless Xposed. I have added many detection and features.
Noticeable updates are:

• Use shell script as updater-binary. Since SuperSU is now a mandatory requirement, using the same format should be acceptable.
• Fixed signed zip unable to unzip issue. This was your main reason to include busybox in installer package. I compared SuperSU's zip and normal zips signed by Signapk.jar and found out that local file headers have missing informations (due to the fact Signapk.jar outputs the zip to stdout). A re-zipping in perl will fix all headers, but signature remains. So there should be no issues to use shell script updater-binaries now
• Added a new mode to build.pl - the "bundle" mode. This mode will create an all-in-one zip for all platforms. Previous single sdk builds are still there and working, so this feature is an addition with no compromises
• Add support for version numbers with decimal numbers in build.pl
• The updater-binary is now multifunctional, features are listed below
• Support both systemless and system install based on detection or user configuration
• User configuration: config files support: /data/.xposed, /cache/.xposed, or /system/.xposed (latter will override previous). Right now the only option is SYSTEMLESS (true/false). If set to false, installer will force system install; if set to true, installer will abort if systemless requirements aren't met
• Detection: If sdk < 23; if system installed SuperSU exist; if system root exist but su.img detected (CM users with manual SuperSU installation); if installed version is 2.76+ when systemless root detected.
• Most detection will only work correctly if data is accessible in recovery (devices with proprietary encryption cannot mount data in custom recoveries). Installer will always use systemless "cache" workaround unless user set SYSTEMLESS=false
• Uninstaller build is added into build.pl. Also, uninstaller can now uninstall both system and systemless installed Xposed. Uses shell script as updater-binary so also cross platform.

[rovo89]

Thanks for your efforts, I really appreciate that you went ahead and discovered the possiblities, when I didn't have the time yet to finalized what I'm already working on.

Unfortunately, I'm afraid that this doesn't go into the direction that I'm willing to take, so I don't think I can use most of your commits. In detail:

Use shell script as updater-binary. Since SuperSU is now a mandatory requirement, using the same format should be acceptable.

I had a lot of compatibility problems before switching to my custom BusyBox executable. There are not that many recoveries and I guess all of them use BusyBox anyway, but I expect the ZIP to work on a running system as well. Having a known, reliable shell with all required commands is a big help and I'm not going to give that up.

Fixed signed zip unable to unzip issue. This was your main reason to include busybox in installer package. I compared SuperSU's zip and normal zips signed by Signapk.jar and found out that local file headers have missing informations (due to the fact Signapk.jar outputs the zip to stdout). A re-zipping in perl will fix all headers, but signature remains. So there should be no issues to use shell script updater-binaries now

It was the main reason why I didn't sign ZIPs originally, yes. Could you provide more information about the corrupted headers please? I think it would be even better to port SignAPK to Perl, I might give that a try...

Added a new mode to build.pl - the "bundle" mode. This mode will create an all-in-one zip for all platforms. Previous single sdk builds are still there and working, so this feature is an addition with no compromises

I see no reason for this and it's not possible without the first change anyway.

Add support for version numbers with decimal numbers in build.pl

I'll have a look at this, but I think it should be fine to merge.

Uninstaller build is added into build.pl. Also, uninstaller can now uninstall both system and systemless installed Xposed. Uses shell script as updater-binary so also cross platform.

Any advantages of integrating it into build.pl?

Regarding the systemless support: I currently don't plan to make that flashable in recovery, as Chainfire advised that there's no officially supported way (by him) to mount su.img. I'm still planning to integrate the installation routines into the flash-script.sh, but things like configuration and /data availibility are probably not needed due to this. The detection part would probably be implemented in Java.

Nevertheless, I'll certainly go through all your changes and see if I find some inspiration.

[topjohnwu]

I had a lot of compatibility problems before switching to my custom BusyBox executable. There are not that many recoveries and I guess all of them use BusyBox anyway, but I expect the ZIP to work on a running system as well. Having a known, reliable shell with all required commands is a big help and I'm not going to give that up.

Actually, I think it is a better idea to install systemless Xposed through installer app. The app can have proper busybox, accessible /data, su.img already mounted, su version detection much easier etc. But I'm no expert in Android app developing so I cannot do it myself.

It was the main reason why I didn't sign ZIPs originally, yes. Could you provide more information about the corrupted headers please? I think it would be even better to port SignAPK to Perl, I might give that a try...

I asked Chainfire how he has achieved signed zip unzipping in all recoveries, he said he used his own tools and didn't give more details. Since the problematic recoveries report header errors in the error messages, I started to investigate with hex comparisons. According to official zip documentations, each file requires a local file header.

local file header signature     4 bytes  (0x04034b50)
version needed to extract       2 bytes
general purpose bit flag        2 bytes
compression method              2 bytes
last mod file time              2 bytes
last mod file date              2 bytes
crc-32                          4 bytes
compressed size                 4 bytes
uncompressed size               4 bytes
file name length                2 bytes
extra field length              2 bytes

file name (variable size)
extra field (variable size)

After viewing signed zips, the general purpose bit flag has been set to different values.
It seems that the problematic unzip cannot support these kinds of format

Bit 3: If this bit is set, the fields crc-32, compressed 
size and uncompressed size are set to zero in the 
local header.  The correct values are put in the 
data descriptor immediately following the compressed
data.

In some where on the internet I found out that if the zipping process is using stdin and stdout, it will not know the file information when constructing the header, so fields crc-32, compressed
size and uncompressed size are set to zero. Seems just like what Signapk.jar does.

I was wondering whether the signing process is something to do with the zip format, or simply just adding the files META-INF/CERT.RSA, META-INF/CERT.SF, META-INF/MANIFEST.MF. After doing some experiments with apks and trying to install them, I found out that it's the latter. Generating the three files are what the signing process actually does. So I simply rezip the zip files in perl, so all headers and the missing fields are back.

[topjohnwu]

Oh, another good reason for using shell script as flashable zip is that it can be called in rom zips.
Some rom devs are asking me to update my version because they can include them in there roms as an option (aroma etc.), just like SuperSU does.

I know you are not supporting rom devs to include them into roms though :)

[topjohnwu]

Regarding the systemless support: I currently don't plan to make that flashable in recovery, as Chainfire advised that there's no officially supported way (by him) to mount su.img. I'm still planning to integrate the installation routines into the flash-script.sh, but things like configuration and /data availibility are probably not needed due to this. The detection part would probably be implemented in Java.

Actually I used Chainfire's script to mount them. The only issue is when data is unavailable (I think this is what CF is concerned about).
I used his own merging script for that. I create a new su.img in cache, and it will be merged into /data/su.img at boot by CF's own script in boot image. You can see more details in my scripts.
I think I've mastered CF's systemless root and uses his techniques to achieve many things lol.

[rovo89]

Actually, I think it is a better idea to install systemless Xposed through installer app. The app can have proper busybox, accessible /data, su.img already mounted, su version detection much easier etc. But I'm no expert in Android app developing so I cannot do it myself.

No problem, I'm already working on it. I just don't have much time, so it takes longer than I had hoped.

Generating the three files are what the signing process actually does.

That, and I think it also adds ZIP comments (which might not be necessary, not sure). Thanks a lot for the information you provided, I'll see if I can change them. As mentioned, prefer even more to avoid SignAPK altogether and just create those three files in Perl. Should be possible.

Oh, another good reason for using shell script as flashable zip is that it can be called in rom zips.

Isn't this the case for my ZIPs as well? Generally, you should be able to execute any flashable scripts by unzipping its update-binary and calling it with proper options as arguments.

[rovo89]

Actually, the comments might be relevant, maybe even more relevant than the three files:
https://github.com/TeamWin/Team-Win-Recovery-Project/blob/b7e8b98ce6641ec5b6a10e91277f3b045e981508/verifier.cpp#L110

[topjohnwu]

What I've done for experiment is I unzipped an APK file and rezip it.
Android can install it without an issue, so I assumed that signature is only involved with the contents, not in the metadata of zip files.

[rovo89]

Are you talking about APKs or flashable ZIP files?

[rovo89]

I have done the research and now I'm sure that repacking the ZIP cannot produce a valid, signed, flashable ZIP file. The three files you mentioned are used for JAR/APK signing, but they're not relevant for flashable ZIP files.

Actually, Google has been working on updating SignAPK in the past weeks and they have even removed the JAR/APK signing part for the -w parameter completely: https://android.googlesource.com/platform/build/+/0caa16a6d1b4349654956c895aab925c9522d2cf

That means only the ZIP file comment with the stored signature is relevant. It's invisible in usual ZIP file viewers because it contains a NULL character after the "signed by SignApk" string. The important point is that this signature includes the whole file, except obviously the file comment and its length. So even if you copy the file comment to the new file, any bit that is modified in the rest of the file will invalidate the signature.

From my point of view, the otacert file and setting the modification times to a fixed date isn't required for usual flashable ZIP, the latter is mainly interesting for incremental OTAs. So all that matters is this signature in the comment, which is mostly independent from the ZIP creation. Unfortunately, I didn't find any Perl modules which can handle PKCS#7/DER/..., but for the use-case of signing with the default test-keys certificate, I have ideas how this could be built in Perl anyway. It's of less priority for me though, first systemless should work fine.

[KaMyKaSii]

Here apparently osm0sis is explaining how Chainfire sign works:

https://forum.xda-developers.com/android/software-hacking/dev-complete-shell-script-flashable-zip-t2934449/post56621542

[0617078820]

B