Working with tarballs in Lite

[bollwyvl]

Assumptions

• work predictably with kernel/DriveFS/contents
• will be fetching non-trivially-sized, CORS-friendly .tar.gz or .zip files
  • or being directed there from the github /tarball/ API

Nice to have

• start downloading/decompressing before a kernel starts
  • better still, be able to be done in parallel while a kernel starts
• be able to pull out a single file from e.g. ?path as fast as possible
  • doing other files as needed, ideally in some pattern

Problems

• working with .zip and .tar(.gz) isn't part of the browser's native JS environment

Payoff

If this approach can be demonstrated here, this would be a solid candidate for eventually moving to its own repo, as this could power other related, pre-kernel-launch capabilities.

Prior art in Jupyter

@hadim/jupyter-archive

this was moved to https://github.com/jupyterlab-contrib/jupyter-archive

jupyter-archive uses a serverextension. While it might be feasible to port this to work on @jupyterlite/pyodide-kernel or the xeus one, it's still a fairly heavy lift to get either kernel started (10s of seconds, still).

@gt-coar/jupyterlab-archive

libarchive.js is an 875kb WASM compilation of libarchive.

Such as it is, the prototype jupyterlab-archive shows a way to do a fairly efficient wrapper around libarchive.js. Further, one could likely do zero-copy transferrable objects.

Is it on npm? tldr: no.

Aside: this was built, pre-JupyterLite, to support archives containing sqlite + assets, so had a companion package, and generally supported ~100mb tarballs. Alas, the examples can't be redistributed, and the effort wasn't meaningfully published... it's unlikely the

If I was starting fresh today, woudl consider the more-maintained, but still 600kb libarchive-wasm.