A proposal for Rye's toolchain management system and Python selection logic

[cnpryer]

Summary

This is a proposal for a revised toolchain management system. There’s a lot to a toolchain management system like this, so this proposal will focus specifically on how Rye utilizes its toolchain management system for its Python version selection logic.

If there’s interest in this proposal I’d like to follow up with completing the same kind of write-up for the design of Rye’s toolchain management system more generally. It’d also be a nice side-effect to be able to help provide some content for #599.

This proposal suggests simplifying how Rye selects Python versions for projects by leveraging its toolchain management system. It should also cover Python selection agnostic of the user’s current project or location on their system. If users opt-out of Rye-managed toolchains we can disregard this logic entirely.

Instead of leaning on .python-version and requires-python in order to determine the version of Python to use, Rye could offer recommended configuration for its Python version selection that’s designed specifically to leverage its toolchain management system. To do this, Rye would expect this configuration set either inside the project or as part of Rye’s home settings.toml file. If no configuration is found, Rye would default to some standard, predictable toolchain such as latest-python.

Motivation

The main motivating factor for simplifying this logic would be to minimize any confusing experiences users might have and make Rye’s toolchain management system as approachable as possible. Toolchains might be a foreign concept to some Python developers. Being simple and intuitive would be important.

Here are some issues that could be considered motivation for this proposal:

• Rye seems to pick "latest fetchable" instead of "latest available" for global shims #586
• Use default toolchain when initializing a project #507
• pyproject.toml bounds checking on existing virtualenv forces oldest Python #437 (suggestion for defaults)
• python version suggestion #335 (default overrides on installation)
• Tox Integration #206 (exposing toolchains)
• Add "python-version" field to pyproject.toml for pinning Python version #96
• How does rye find global python? #489
• Controlling the python version without live prompts? #597
• Using rye to manage global Python installtions like nvm #70
• Rye `pin` vs `use` #303

Currently Rye will check for a .python-version. It’ll fallback to requires-python configuration if .python-version can’t be resolved. Finally a default will be resolved if possible. In addition, Rye offers toolchain configuration via config.toml. This involves a recommended selection path that might end up confusing to some users.

Also, requires-python being part of that selection logic might only add to that confusion. While it could make sense as a fallback, requires-python is defined in PEP 0345 as the “field [that] specifies the Python version(s) that the distribution is guaranteed to be compatible with.” This is relevant to downstream consumers of the distributed package. It’s useful information for determining what Python version to use to develop the project, but it’s inference that may not be necessary for Python version selection logic. It's likely valuable for distribution validation logic, but it might be only adding to the complexity of the current Python selection process. So its value as part of the Python selection process should be evaluated by its balance with the toolchain management system's ease-of-use.

Similarly, .python-version — a file used by some tools as part of Python version selection logic — might seem like a great fit for Rye, but if users are expected to be introduced to a toolchain management system, we should make the introduction process to Rye’s toolchains as simple as possible. Supporting .python-version is fine, but including it as part of that initial experience can be confusing.

As a user of Rye, my first impression was that .python-version is the first-class mechanism used for selecting the Python version. A user's initial interaction with Rye’s Python selection process should be obvious and specifically through its toolchain management system.

Details

Rye offers a toolchain management system. So Python selection should defer to toolchain management behavior.

Instead of what seems like multiple recommended methods for configuring its Python selection process, we can simplify this for users by recommending direct toolchain selection. Here are a few examples of how this could look:

Selecting your toolchain via TOML configuration

[tool.rye]
toolchain = “3.12”    # ([email protected])

The same configuration can be provided in its config.toml file, but I’d suggest recommending standard pyproject.toml tool table configuration to reduce configuration sprawl. A separate config.toml file can be supported as well.

Selecting your toolchain via CLI

rye tc use 3.12    # (rye toolchain use <version>)

Selecting via the CLI could enter the current scope into a settings.toml file under Rye’s home directory and/or enter the configuration into the project’s current configuration file. I’d suggest requiring in-project toolchain configuration to be more manual while the CLI selection can enter the scope into the settings.toml without getting in the user’s way.

Defaults and fallback behavior

With a settings.toml file we can support default overrides for the system. If no override is entered to the settings.toml file we can either use next-best scope or a default for latest-python (or latest locally available toolchain).

Using a settings.toml file in Rye’s home directory will also provide Rye with options for exposing toolchains globally. We also are able to think about how this design can support better exposure for other tools like tox.

Lastly, I really like the concept of channels. Beyond providing better definition for toolchain defaults and Rye’s selection process, channels can give us options for more dynamic methods for labeling toolchains or distribution sources.

Settings.toml and channels

This draws inspiration from Rust's Rustup settings.toml configuration.

• Multiple projects can have entries in a settings.toml.
• Allow for a configured default (or default overrides).

We can also define the toolchain management system in a way where it’s Rye-agnostic. To do this, we can define toolchain directories as toolchain installation targets. We can standardize these directories to contain their own settings.toml files instead of adding this data to the home directory’s settings file.

[scope]
"/" = "default"  # Resolve to "default" for any paths under the root (sets "default" as default)
"some/project/path" = "toolchain path"

If we want to define the version selection as something a little more dynamic we can use channel names. Channel names would be useful for Rye if we intend to provide Rye with more toolchain sources (or any kind of additional distribution management).

rye toolchain use <channel>           # Use a toolchain via its channel

For example, channels might be helpful for differentiating distribution kinds.

[mitsuhiko]

I still need to digest this, but there is another change pending now that relates to toolchain management. #606