generate a multi-level PyPi index as per PEP 503

[tdyas]

PEP 503 describes a multi-level PyPi index with a root index file with just links to the supported "projects" and project-specific index files with the actual links to release files.

Update generate_index.py to generate a multi-level index and modify the test accordingly.

[tdyas]

I tried to solve with dumb-pypi, but that tool cannot handle asset URLs which do not share the same prefix.

[huonw]

Makes sense to me, thanks for picking it up

[huonw]

My read of pep-0503 is that the URLs must always be exactly /<project>/, i.e. customising this --url-path-prefix value may not work at all, and so potentially shouldn't be exposed?

[tdyas]

My read of PEP 503 is that they are the full path with trailing /. And checking PyPi itself shows that the links include the prefix since they are /simple/PKG_NAME/. (For example, look at curl https://pypi.org/simple/ -o pypi.html.)

[huonw]

PEP 503 seems to suggest we MUST be using the normalised names in various places. Does this guarantee name is the normalised package name? Or maybe the wheel file names are guaranteed to be normalised?

[tdyas]

Yes, the normalized name is returned: https://packaging.pypa.io/en/stable/utils.html#packaging.utils.parse_wheel_filename

[benjyw]

Curious what prompted this?

[tdyas]

Curious what prompted this?

Slack request from a user: https://pantsbuild.slack.com/archives/C046T6T9U/p1732700284097479

[tdyas]

I pulled the test out to #2 and added a GitHub Actions workflow for it. Once that other PR lands, I'll update this PR accordingly.

[tdyas]

@huonw / @benjyw: scie-pants has a reference to the single index file on https://wheels.pantsbuild.org/simple/. https://github.com/pantsbuild/scie-pants/blob/be453bb700dfe1728efdc41a68c41c266144bc58/tools/src/scie_pants/pants_version.py#L77

Is this something which has the potential to break if this PR lands?

[tdyas]

Back to draft to prevent landing this PR until after we determine any effects on scie-pants.

[huonw]

Ah, hm, good question. I suspect it's probably fine for new versions of scie-pants: a few lines later it is suggested that code path is only used for pants < 2, and I think this index file only supports pants >= 2, so the wheels referenced here shouldn't be useful.

https://github.com/pantsbuild/scie-pants/blob/be453bb700dfe1728efdc41a68c41c266144bc58/tools/src/scie_pants/pants_version.py#L81

Things I don't know:

• if changing this will cause modern scie-pants to start crashing when installing very old Pants versions
• if there's older still-in-use versions of scie-pants for which this might actually matter functionally (i.e. these find-links are actually used)

[tdyas]

Ah, hm, good question. I suspect it's probably fine for new versions of scie-pants: a few lines later it is suggested that code path is only used for pants < 2, and I think this index file only supports pants >= 2, so the wheels referenced here shouldn't be useful.

scie-pants supports installing Pants v1? That seems odd to me.

[huonw]

Yeah, in theory it does: one-stop-shop for running all Pantses. There's even a test of running 1.30.5rc1: https://github.com/pantsbuild/scie-pants/blob/be453bb700dfe1728efdc41a68c41c266144bc58/package/src/test.rs#L830-L851

In practice, just now, I've not been able to get this running locally (e.g. in a x86-64 linux docker container) due to errors like: ERROR: Could not build wheels for thriftpy2, which is required to install pyproject.toml-based projects.

Installing pantsbuild.pants==1.30.4 into a virtual environment at /root/.cache/nce/45436524b7fa68f7a41eb7ac53bc7bf9441def442ddb1ac1d5610
4ae54ed442d/bindings/venvs/1.30.4
  Installing build dependencies ... done
  WARNING: Missing build requirements in pyproject.toml for thriftpy2>=0.4.0 from https://files.pythonhosted.org/packages/f8/3a/d983b26d
f17583a3cc865a9e1737bb8faacfa1e16e3ed17353ef48847e6b/thriftpy2-0.5.2.tar.gz (from py-zipkin==0.20.0->pantsbuild.pants==1.30.4).
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  error: subprocess-exited-with-error
  
  × Building wheel for thriftpy2 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for thriftpy2 (pyproject.toml) ... error
  ERROR: Failed building wheel for thriftpy2
ERROR: Could not build wheels for thriftpy2, which is required to install pyproject.toml-based projects

[notice] A new release of pip available: 22.3.1 -> 24.3.1
[notice] To update, run: python -m pip install --upgrade pip

The version of pip mentioned there seems to come from https://github.com/pantsbuild/scie-pants/blob/be453bb700dfe1728efdc41a68c41c266144bc58/tools/src/scie_pants/install_pants.py#L70

Not sure why there's the difference between scie-pants CI (which is passing) vs. running locally.

I've posted pantsbuild/scie-pants#431 to simulate this change and see what scie-pants' CI says.

[benjyw]

I'd definitely like to understand the full implications before merging. Can we not generate both indexes? The new one can go to some new location, since people who want to use it are aware of it. That said, scie-pants could perhaps also benefit from the faster lookup?