RFE: functionality to set __version__ from scm automatically

[jonathanunderwood]

The setuptools world has a module setuptools_scm which allows setting the version directly from the SCM in use for the project. For example, if a commit is tagged with a version identifier, that becomes the value of version, whereas if the commit is not tagged, the hash and the distance from the last tag may be used.

This is super useful, and often critical, functionality for automation. It would be great if flit had machinery for doing this kind of thing.

[Freso]

I fully agree, but flit’s author does not seem to. A previous related request was turned down and also a PR implementing this (not because of how it was implemented, but purely the idea of it). 😞

[jonathanunderwood]

Wow, that's very sad indeed. But it's helpful to know while I'm evaluating flit and poetry.

[takluyver]

I'm open to thinking about ways to make it easier to manage version numbers. But the sort of solutions people have proposed so far all involve extra complexity inside every package to look up its version number when it gets imported. That's what I object to. Any extra complexity needed should go in developer tooling, not in the packages' runtime code.

[jonathanunderwood]

But the sort of solutions people have proposed so far all involve extra complexity inside every package to look up its version number when it gets imported.

I haven't had time to review the previous proposals. But, at least in the case of setuptools_scm, very little is required of the developer: (1) require setuptools_scm (wouldn't be needed if it was a capability directly in flit); (2) Optionally define a file for the version tag to be written to - this is only needed if you don't want to make use of the pkg_resources functionality for finding the package version at runtime.

So, I'd hope it was possible for flit to support this something like:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"
version-backend = "flit.vcs_version"

Would you regard that as requiring too much of a packager?

[takluyver]

I'm not concerned about what it involves for the developer. I'm concerned about what the package winds up doing at runtime. It needs some extra dependency installed, and then it goes hunting for its metadata to figure out it's version number. I consider this similar to an object using introspection to look up its variable name in the call stack: you can do it, but you shouldn't. It can also have performance issues, because it winds up having to scan every installed package to find the one that it is. And it definitely violates the principle of 'do as little as possible on import'.

I definitely want version numbers to be 'stamped' into the code of a package somehow so the package doesn't need to go looking for them. I tend to do that manually. I'm aware of tools like bumpversion to partially automate it, but so far I haven't really taken to any of them.

[Midnighter]

Just to add my voice that this is stopping me from using flit for my build tool chain. A pity really. versioneer (sadly no longer supported) installs a module into the package to do the parsing whereas setuptools-scm use pkg_resources (maybe a bit heavy). Ultimately, both write the version string at the time of distribution into PKG-INFO and parse it from there for installed packages.

Do you consider this behavior unacceptable? Especially the versioneer way of installing a module into the package?

[jonathanunderwood]

Just to add my voice that this is stopping me from using flit for my build tool chain. A pity really. versioneer (sadly no longer supported) installs a module into the package to do the parsing whereas setuptools-scm use pkg_resources (maybe a bit heavy).

Just to note: setuptools_scm doesnt have to use pkg_resources. It can simply write the version to a file that is then included in the package.

Concerns about using pkg_resources are orthogonal to the issue here.

[Midnighter]

My bad, indeed setuptools-scm writes the version to PKG-INFO but the recommended way (from the README) for accessing the version in the installed package is to use pkg_resources:

from pkg_resources import get_distribution, DistributionNotFound
try:
    __version__ = get_distribution(__name__).version
except DistributionNotFound:
    # package is not installed
    pass

Concerns about using pkg_resources are orthogonal to the issue here.

While I tend to agree with your statement, I interpret @takluyver's comments in a way that these are exactly the concerns that he has. Specifically, I'm referring to this part:

But the sort of solutions people have proposed so far all involve extra complexity inside every package to look up its version number when it gets imported. That's what I object to. Any extra complexity needed should go in developer tooling, not in the packages' runtime code.

[uranusjr]

Random idea by a passer-by: Would it be a good idea to have such a configuration:

# mypackage/__init__.py

__version__ = "__flit_scm_version__"

When flit builds the sdist and wheel, it would detect this special version token. If it is detected, parse __init__.py with ast and replaces sources in artifacts with the correct version. This would solve the ditribution workflow problem preferred by setuptools-scm user, but without compromising the version detection mechanism for end users.

Some downsides I can think of:

• Users gets a nonsensical value if the package version is read without being installed. This is probably minor though. setuptools-scm’s suggested solution is only marginally better IMO. Or maybe we can go with __version__ = __flit_scm_version__ which would raise a NameError at runtime, but requires some extra configuration to keep linters happy.
• This won’t work with flit install -s (while setuptools-scm works well with setup.py develop). Maybe pth wold work with some hacks though?

[Zaharid]

Some downsides I can think of:

* Users gets a nonsensical value if the package version is read without being installed. This is probably minor though. setuptools-scm’s suggested solution is only marginally better IMO. Or maybe we can go with `__version__ = __flit_scm_version__` which would raise a `NameError` at runtime, but requires some extra configuration to keep linters happy.

* This won’t work with `flit install -s` (while setuptools-scm works well with `setup.py develop`). Maybe pth wold work with some hacks though?

The __version__ could be made to be stamped at install time (either in __init__ or rather in a separate file) but looked at dynamically (e.g. calling git describe) at runtime. The package author would write:

# __flit_version__.py

import some_package_to_get_versions
__version__ = some_package_to_get_versions.get_version()

And

#__init__.py
from . __flit_version__ import __version__
...

At install time flit would rewrite __flit_version__.py to contain only whatever the value of __version__ (on import). It wouldn't have to know about how it works inside.

#__flit_version__.py (rewritten)
__version__  = "<String with the actual version>"

While the version would be resolved dynamically from SCM (or whatever) when the package is symlinked for development.

One workflow where it is not trivial to change versions manually every times is if one wants to publish to PyPI every time a change is made to the master branch, using some CI service. The packages are then rejected because they are duplicated.

[anarcat]

i am using setuptools_scm and the way that works is basically this:

in setup.py:

setup(
    use_scm_version={'write_to': 'MODULE/_version.py'},
    # [...]
)

Then in the module's __init__.py:

try:
    from _version import version
except ImportError:
    try:
        from setuptools_scm import get_version
        version = get_version()
    except (ImportError, LookupError):
        version = '???'
__version__ = version

It seems to me flit could build a wheel the same way, by dropping a magic file at build time so that, at runtime, the version would be available.

The alternative is to load the version through the entrypoint. That has traditionally been slow (because of pkg_resources, see pypa/setuptools#510) so maybe that should be avoided.

In general, this smells like PEP material to me: there should be a standard way for Python packages to fetch their own version, and a standard way for Python packaging tools to write that version somewhere.

In any case, it seems to me that having to maintain that version in two places is counter-intuitive and something I would like to avoid, because it's bound to create releases with the wrong version number metadata.

It is also the primary reason why I haven't used flit in this new project.

Thanks for your work!