Fix uncaught TypeError accessing setuptools_dist.version

[ctcjab]

The latest version of pip is missing some defensive code when trying to populate the User-Agent header and the setuptools_dist.version is not as expected, leading to errors like the following:

38.33 ERROR: Exception:
38.33 Traceback (most recent call last):
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
38.33     status = _inner_run()
38.33              ^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
38.33     return self.run(options, args)
38.33            ^^^^^^^^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
38.33     return func(self, options, args)
38.33            ^^^^^^^^^^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 325, in run
38.33     session = self.get_default_session(options)
38.33               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/cli/index_command.py", line 76, in get_default_session
38.33     self._session = self.enter_context(self._build_session(options))
38.33                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/cli/index_command.py", line 99, in _build_session
38.33     session = PipSession(
38.33               ^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/network/session.py", line 344, in __init__
38.33     self.headers["User-Agent"] = user_agent()
38.33                                  ^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/network/session.py", line 179, in user_agent
38.33     data["setuptools_version"] = str(setuptools_dist.version)
38.33                                      ^^^^^^^^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py", line 168, in version
38.33     return parse_version(self._dist.version)
38.33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_vendor/packaging/version.py", line 56, in parse
38.33     return Version(version)
38.33            ^^^^^^^^^^^^^^^^
38.33   File "/opt/conda/lib/python3.12/site-packages/pip/_vendor/packaging/version.py", line 200, in __init__
38.33     match = self._regex.search(version)
38.33             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
38.33 TypeError: expected string or bytes-like object, got 'NoneType'

This looks similar to #11352.

One example of where this occurs is when attempting to pip install a package in the "base" conda environment in the docker images provided by https://github.com/jupyter/docker-stacks. This occurs both with the latest release version of pip installed in the environment, as well as the latest development version. While this issue is unaddressed, I am working around it by reinstalling setuptools in this environment, which allows subsequent pip install invocations to succeed.

This patch addresses the issue by catching the TypeError and skipping the population of the setuptools_version in this case. Better to allow installation to continue with a User-Agent header that is missing the setuptools version than to hard fail just because of some missing telemetry.

In that spirit, perhaps there should be a broader try/except Exception around the entire self.headers["User-Agent"] = user_agent() call, but I'm starting with the most minimal patch that would address the issue.

If any changes are required to this patch before it will be merged, I will do my best, but unfortunately can't promise anything more than this initial submission. I hope this is helpful in any case.

[pfmoore]

This sounds like your copy of setuptools is not installed correctly, lacking the necessary installation metadata. Is it possible that the setuptools install was done by conda, and conda isn't following the standard for how to record package metadata?

I appreciate that a failure in this situation is annoying, but it has demonstrated that there's a problem with the underlying installation, and I'm inclined to say that the correct thing to do is for you to fix that problem, rather than for pip to simply ignore it.

[ctcjab]

Thanks for the quick reply.

Is it possible that the setuptools install was done by conda, and conda isn't following the standard for how to record package metadata?

I'm not sure what resulted in the setuptools metadata in this environment getting into this state. But as I mentioned, running conda install setuptools to reinstall setuptools in this environment before invoking pip works around this issue, so that demonstrates that conda is following the standard in that case.

Something like this ought to provide a minimal repro (untested):

FROM jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa

# Uncomment out to work around the bug triggered by the pip install below
# RUN conda install setuptools

# Repro
RUN /opt/conda/bin/pip install "faiss-cpu<=1.8.0"

I'm inclined to say that the correct thing to do is for you to fix that problem, rather than for pip to simply ignore it.

While we don't yet know how the setuptools installation has gotten into this bad state and how many users it could be affecting, it seems to me that it would be better for pip users if pip proceeded to install the package the user needs, which is the user's goal, and at most warned about the setuptools installation, rather than hard-failing just because it couldn't include the setuptools version in the User-Agent header, which is not the user's goal. If failing to find the setuptools version will definitely cause other problems that will get in the way of the user's goal, it would be another story, but I'm not sure that's the case. We've been using this base image's base conda environment (without first reinstalling setuptools in it) to serve jupyter lab to our users for a while now with no problems. We only started to hit this recently upon conda installing jupyterlab-ai in that environment (due to its questionable practice of calling pip install in a post-link script).

[notatallshaw]

I can't reproduce your error, the dockerfile fails to build:

 => ERROR [internal] load metadata for docker.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f  0.4s
 => [auth] jupyter/minimal-notebook:pull token for registry-1.docker.io                                                                       0.0s
------
 > [internal] load metadata for docker.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa:
------
pip-error.dockerfile:1
--------------------
   1 | >>> FROM jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa
   2 |
   3 |     # Uncomment out to work around the bug triggered by the pip install below
--------------------
ERROR: failed to solve: jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa: failed to resolve source metadata for docker.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa: docker.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa: not found

Without the hash is builds fine:

[+] Building 5.2s (6/6) FINISHED                                                                                                    docker:default
 => [internal] load build definition from pip-error.dockerfile                                                                                0.0s
 => => transferring dockerfile: 245B                                                                                                          0.0s
 => [internal] load metadata for docker.io/jupyter/minimal-notebook:latest                                                                    0.0s
 => [internal] load .dockerignore                                                                                                             0.0s
 => => transferring context: 2B                                                                                                               0.0s
 => [1/2] FROM docker.io/jupyter/minimal-notebook:latest                                                                                      0.3s
 => [2/2] RUN /opt/conda/bin/pip install "faiss-cpu<=1.8.0"                                                                                   4.5s
 => exporting to image                                                                                                                        0.4s
 => => exporting layers                                                                                                                       0.4s
 => => writing image sha256:a8981e1a7b40ce41316fb81d3c48eed51906f5938966866fa328e4c52f831995                                                  0.0s

[ctcjab]

I forgot the "quay.io".

FROM quay.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa
# ...

❯ docker build .  # succeeds
...
=> [internal] load metadata for quay.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa                                                                                                                                                                        0.2s
...
 => exporting to image                                                                                                                                                                                                                                                                                           0.0s

[notatallshaw]

Builds but doesn't reproduce the error:

[+] Building 5.4s (6/6) FINISHED                                                                                                    docker:default
 => [internal] load build definition from pip-error.dockerfile                                                                                0.0s
 => => transferring dockerfile: 325B                                                                                                          0.0s
 => [internal] load metadata for quay.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa     0.0s
 => [internal] load .dockerignore                                                                                                             0.0s
 => => transferring context: 2B                                                                                                               0.0s
 => [1/2] FROM quay.io/jupyter/minimal-notebook@sha256:67fcff2665c371ffa29b78255bb0591672c5ca20487df0887f55685cd6f9b7aa                       0.3s
 => [2/2] RUN /opt/conda/bin/pip install "faiss-cpu<=1.8.0"                                                                                   4.7s
 => exporting to image                                                                                                                        0.4s
 => => exporting layers                                                                                                                       0.3s
 => => writing image sha256:d92b0849dfab48d3a82faea52056cab77298739827ce5fc15f6165cf2d6020b5                                                  0.0s

While we don't yet know how the setuptools installation has gotten into this bad state and how many users it could be affecting, it seems to me that it would be better for pip users if pip proceeded to install the package the user needs, which is the user's goal, and at most warned about the setuptools installation

The problem seems deeper than "I can't generate a part of a user agent string". The version of an allegedly installed distribution is throwing an error, lots of operations in pip require knowing the versions of installed distributions, coding defensively here seems like it would make the error pop up later down the line.

[ctcjab]

Was hoping I'd correctly eyeballed a minimal repro, but alas. And I can't share the unminimized code where this reproduces since it's proprietary. I'm afraid I won't be able to take this farther any time soon. Feel free to close if you disagree with the patch and thanks for your consideration.