Add Python project definition and pip smoke test

[thillux]

Closes #5038.

[thillux]

@arckoor

[randombit]

So I'm open to this in general but I'm confused about how this, as it stands, would be used in practice. It seems like the pyproject.toml is not being installed anywhere, so how is the user expected to get it?

Also while I'm not sure where pyproject.toml should go, it should definitely not be in the top level directory.

[thillux]

I see multiple options here:

1. This can probably be used to publish botan3 to PyPI IMHO. But this should be at your discretion. People already started to provide wrappers for botan there, as they can't find it. https://pypi.org/project/slh-dsa-botan3/
2. I'll use this to add botan3 as python package to nixpkgs (current draft without open PR here: thillux/nixpkgs@135929c).

Regarding the position of the file in the repository: just tell me your preference, no fixed opinion on my side.

[thillux]

I'll test moving this file to src/python.

[thillux]

Moving pyproject.tomlto another path (src/python) does not work, as long as files above this path shall be referenced. In this case readme.rst and license.txt where not accessible.

        File "/tmp/nix-shell-26044-0/nix-shell-26201-0/pip-build-env-kmzfd5o0/overlay/lib/python3.13/site-packages/setuptools/config/expand.py", line 129, in <genexpr>
          if _assert_local(path, root_dir)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
        File "/tmp/nix-shell-26044-0/nix-shell-26201-0/pip-build-env-kmzfd5o0/overlay/lib/python3.13/site-packages/setuptools/config/expand.py", line 149, in _assert_local
          raise DistutilsOptionError(msg)
      distutils.errors.DistutilsOptionError: Cannot access '/home/mtheil/Code/botan/src/python/../../readme.rst' (or anything outside '/home/mtheil/Code/botan/src/python')
      [end of output]

If these lines are removed, moving the file is possible.

$ pip wheel src/python --no-deps
Processing ./src/python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: botan3
  Building wheel for botan3 (pyproject.toml) ... done
  Created wheel for botan3: filename=botan3-3.8.2.dev330+ge7433da4f.d19800101-py3-none-any.whl size=17415 sha256=e87f8d9759a026fb20abf4fc79c54497e8ad92e745557584bb8b6d9fc613057e
  Stored in directory: /tmp/nix-shell-26044-0/nix-shell-26201-0/pip-ephem-wheel-cache-t666_cmq/wheels/9d/76/33/70c4063eda8a042c9aa14ab1470b961a913b789ffc085ea1c5
Successfully built botan3

[coveralls]

coverage: 90.676% (-0.004%) from 90.68%
when pulling a091aaa on thillux:mtheil/python-definitions
into feb9b3f on randombit:master.

[thillux]

@randombit moved to src/python and removed non-accessible references.

[acid-bong]

Implements #5038.

To make the issue close on merge, change the wording to fixes ### or closes ###

[arckoor]

I would be in favor of placing the pyproject.toml in the top level. All the python scripts are to my knowledge completely hand-formatted, which makes them a bit of a mess. Placing the toml in the top level would allow configuring tools like black there.
I'm very fond of black as a formatter, it does default to a max line length of 88 though, which for me is a little short.
So with the toml one could do

[tool.black]
line-length = 120

[reneme]

@thillux @arckoor Do you think it would make sense to vendor a prebuilt botan3.so (and likewise for other platforms) within the python package? In the hope that pip install botan3 just sets everything up properly. Or is such a thing generally a bad idea?

If I understand the situation correctly, on Ubuntu I would otherwise have to pip install botan3 and also apt install botan. Only to realize that Ubuntu would likely ship a too-old version of the shared object.

[arckoor]

For distribution via pip I think it would definitely make sense, but should somehow be optional? I.e. I consume basically all my python packages through nixpkgs, and there a prebuilt .so just won't work iiuc, so what's done there is the underlying botan derivation is built and that is passed to botan3.py. (Don't quote me on that, I've only consumed nixpkgs so far, I'm not that privy to its inner workings :p)

The lupa package has a similar usecase, and they include a prebuilt version in their wheel, so I think something like that would be the way to go here too?

[bkmgit]

A PyPI package is helpful. For distributions, being able to use the system provided botan shared library instead of creating another copy is better.

[thillux]

@reneme No hard opinion on the inclusion of botan as shared object. IMHO I'd prefer to get that from my Linux distribution outside of the pip package.

[bkmgit]

This is really helpful. Makes packaging for Fedora much better:
https://copr.fedorainfracloud.org/coprs/build/9760605

Have the pyproject.toml file at the top level is nice, but it is possible
to have configure.py
move files around and control creating a python wheel.

Rebased on current head of repository:
https://github.com/bkmgit/botan/tree/python-pyproject

Happy to add tooling to be able to publish to PyPI and produce wheels.

[bkmgit]

This would help resolve #5103