Micropip.freeze no work? #107
Comments
|
Indeed we need some more robust test for micropip.freeze, especially its compatibility with pyodide-lock (#88). By the way, is it possible to install fastapi in Pyodide, without skipping some packages? When I try to install it in the browser, I get: Can't find a pure Python 3 wheel for 'ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1'. |
|
Are you on latest? It works in pip. |
|
@jaraco this might be a good first issue. |
|
I'm getting started with this issue. Attempting to follow the repro instructions, I first don't want to run Next, I tried to create the virtualenv, but it fails: It wasn't obvious to me at first the command had failed. The message
looks more like a warning or informational message than an error. This does remind me that I need to work on my shell prompt to show when the last command was a failure. So it seems I need to install pyodide using Python 3.11. I used pipx to uninstall and then reinstalled with After installing the virtualenv, I find that it doesn't honor pylauncher as I'd expect. Pylauncher does launch the Python in the virtualenv: But when invoking pip, it seems to try to target the system environment: The way pylauncher works is it's supposed to use but for some reason pip doesn't honor that expectation. That may be a separate issue deserving its own investigation. In the meantime, I'll invoke pip explicitly. Edit: Invoking Invoking pip and python explicitly, I'm able to replicate the failed expectation: |
|
Attempting to editable-install micropip to the target environment, I encountered another issue. The metadata fails to uninstall due to pip not finding the required 'name' metadata. |
|
If I apply this diff: micropip main @ git diff
diff --git a/micropip/_commands/freeze.py b/micropip/_commands/freeze.py
index 4a49dc5..6c2804a 100644
--- a/micropip/_commands/freeze.py
+++ b/micropip/_commands/freeze.py
@@ -26,11 +26,7 @@ def freeze() -> str:
name = dist.name
version = dist.version
url = dist.read_text("PYODIDE_URL")
- if url is None:
- continue
-
sha256 = dist.read_text("PYODIDE_SHA256")
- assert sha256
imports = (dist.read_text("top_level.txt") or "").split()
requires = dist.read_text("PYODIDE_REQUIRES")
if not requires:FastAPI then appears in the output: |
|
At first blush, things seem to be working as intended. The docs for
But since the fastapi package wasn't loaded with Then again, I don't yet understand what micropip is or what interactions it has with pip. Clearly many of the packages are getting pyodide-specific metadata, so I need to learn how it is that micropip is affecting a pip install. |
|
I read the readme and docs, but they don't explain how micropip works except from a user standpoint, so I'm starting to inspect the code and the environment to understand how pyodide/micropip works. I can see that the file at @hoodmane Can you direct me to where the behavior deviates from the default |
|
Oh! I just noticed that none of the packages installed by pip are showing up in diff --git a/micropip/_commands/freeze.py b/micropip/_commands/freeze.py
index 4a49dc5..23b4bbb 100644
--- a/micropip/_commands/freeze.py
+++ b/micropip/_commands/freeze.py
@@ -28,6 +28,8 @@ def freeze() -> str:
url = dist.read_text("PYODIDE_URL")
if url is None:
continue
+ else:
+ breakpoint()
sha256 = dist.read_text("PYODIDE_SHA256")
assert sha256That is, not a single distribution has the expected metadata. The sole output of So that begs the question - what isn't working? What is it you expect from |
|
We also have https://github.com/pyodide/pyodide-lock, so maybe this "create a lock file from the pip environment" could go there. The problem is that micropip is not a good solve algorithm, and pip is unable to create the lockfile we want. So it makes sense to try and make the two get along so we can get the solve from pip and the lockfile from micropip. |
|
Ah yes the logic does seem to be rather intentionally ignoring externally installed packages. Maybe we can try a bit harder here. In this situation we have two indexes.
Our jsdelivr index is all in |
|
But if it did come from pypi and we know which version it is then we have |
|
Is there a spec or description of what the lockfile requirements are? I see in pyodide_lock.spec, the PackageSpec indicates that file_name must be a string. Could that string be empty? Does it need to exist? What would fail if it doesn't exist? What would happen if it points to a file that doesn't actually match the installed package? |
|
There is uh no written spec. We should write one and probably add it somewhere to the Pyodide packaging docs. The situation with
Then people were installing wheels from other sources with micropip using its slow and incorrect resolver, I was upset by this and wanted a way to lock additional wheels. Because of the way So after this the field contains either a file name if it's from the jsdelivr cdn or a fully qualified url. Working from an implementation detail of the installer. We should clean this up. |
String cannot be empty in that case it will fetch the directory and jsdelivr will return a 404 or someone else's server may return some directory index. This is a special case of it not existing. Depending on what exactly happens, either the fetch failing leads to the install failing, or we get an html response or some other thing that is not a zip archive and we try to unzip it and fail in unzip.
If it's a wheel that is something different, I am not sure. Sounds like an excellent test case! Most likely it just installs the wrong thing? |
Neat. Although it's probably unintended, I'm happy to see URL joins working the way they were intended to do so, allowing files to be resolved from a default location or from somewhere halfway around the world, on a whim. Thanks for the explanation. I think I understand the problem pretty well now. My first instinct is to compare these lock files against other lock files created in the Python ecosystem, such as with In the case of pip freeze, pip stores the name and version, but not the URL or hash. Piptools compile saves the name and version and hash, but not the URL (IIRC). Both approaches rely on the installer and defer resolving the URL to the installer at install time. Before we go to the trouble of implementing something that locks in the URL and hash, I'd like to ask ourselves if it might make sense to follow that model and require pyodide (or whatever installer consumes the lockfile) to use infrastructure (PyPI) to resolve the asset? Imagine, for example, that the lockfile has these fields: Then load-package.js could honor the This approach would allow freezing an environment with packages installed by pip or other standards-conforming installer. It's a bit of a shame that PEP 610 specifies that direct_url.json is only for installs from a URL that weren't resolved through an index. So as I see it, we have three main avenues to pursue:
Are there other factors to consider when choosing from these options? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The text was updated successfully, but these errors were encountered: