Dependencies resolved but installation fails because of incompatible package versions for Python 3.10

[venaturum]

(using Poetry 1.2.1)

I have a project which I publish to pypi, which has the following dependencies

[tool.poetry.dependencies]
python = "^3.6.1"
pandas = "^1"
numpy = "^1.14"
matplotlib = ">=2"

I can pip install my project with python 3.10 - it works fine. However I can't seem to get a poetry environment for development to install properly. The steps I have taken are:

1. delete lock file
2. remove any existing 3.10 environment
3. poetry env use <path to 3.10 executable>
4. poetry install

(These steps work perfectly for python 3.6 to 3.9)

With 3.10 resolution works, but during installation it complains about the numpy version.

So I changed the numpy dependency to this:

numpy = [
    {version = "^1.14", python = ">=3.6.1,<3.10"},
    {version = "^1.21.2", python = ">=3.10"}
]

Once again it resolves ok, but then complains about matplotlib. So I update those dependencies and repeat. Next it complains about pandas. So I update those dependencies. By now the dependencies look like this

[tool.poetry.dependencies]
python = "^3.6.1"
pandas = [
    {version = "^1", python = ">=3.6.1,<3.10"},
    {version = "^1.3.4", python = ">=3.10"}
]
numpy = [
    {version = "^1.14", python = ">=3.6.1,<3.10"},
    {version = "^1.21.2", python = ">=3.10"}
]
matplotlib = [
    {version = ">=2", python = ">=3.6.1,<3.10"},
    {version = "^3.5", python = ">=3.10"}
]

Once again it resolves ok, but then complains about scipy version during install. Scipy isn't even one of my direct dependencies - do I have to now add it as one to fix this issue? Why is the resolution process producing "solves" which won't work?

Any help would be appreciated. I've sunk about 10 hours into trying to sort this out.

[neersighted]

It's hard to understand what is going on on your machine without logs -- however, on my machine, Numpy must be built from source on 3.10 as 1.19.5 is selected and there are no 3.10 wheels for that version. I suspect in general you are fighting wheel availability -- logs would be welcome to further debug this. However, keep in mind that in general Poetry will pick the most compatible version of a dep (that is as new as possible), regardless of available wheels, which it does not consider a factor in solving.

If you care about the version of a transient dep (because of wheels) you need to tell Poetry that by adding a constraint. You can use a group to categorize and sort your dependencies.

[neersighted]

(as to why this works with pip, Poetry has an exhaustive resolver and solves for all versions at once. pip is picking newer versions than Poetry -- some may be partially incorrect, but mostly it will only solve for 3.10 and thus pick versions that all have wheels. Increasing your constraints across the board or using multiple constraints like you have already started is the way to solve this)

[venaturum]

Thanks Bjorn. I think I understand. In these multiple constraints I have used above, version 3.10 is the special case. Would it be best practice to define version numbers for every minor Python release (and should those version numbers have upper limits)?

I'm also not 100% sure how to define this - the example in the docs is perhaps a bit too simplistic and I think we could use a harder example. I can put the below into an Issue if it helps?

The image below shows the dependency matrices for the "foo" example in the docs, and also for numpy. In the foo example the matrix is composed of two non-overlapping sub-matrices - i.e. if we created a graph with version nodes and edges for compatibility then we would have two unconnected sub-graphs. This is not the same for the numpy case and so how to define this with multiple dependency constraints is ambiguous and would either be:

a) This numpy version is compatible with these Python versions, or
b) This Python version is compatible with these numpy versions

I assume it is b) - which is what is shown in the image below - but I need confirmation on this (and would you recommend that I add this to the pyproject.toml file)?

[neersighted]

I'm not following 100% right now (I don't really understand what a vs b actually means), but looking at your example matrix, that looks quite reasonable. Much of your pain is coming from the fact that NumPy doesn't specify a upper bound on compatibility, and Python API changes break compiling it on newer versions.

It would be awesome if you wanted to contribute this more complex/non-toy example to the docs -- NumPy specifically causes this to be necessary and is a good real world example of when you need to use multiple constraint versions.

[venaturum]

if you wanted to contribute this more complex/non-toy example to the docs

Yeh that's where I was headed with this.

The difference between a) and b) comes from how we link multiple python versions to multiple package versions. a) links multiple python versions to one package version, and repeats for each package version. b) links multiple package versions to one Python version, and repeats for each python version. It's analogous to representing a 2D matrix as a list of lists. The inner lists are either lists of Python versions, or lists of package versions.

I probably need confirmation before proceeding.

[neersighted]

Ah, I see. I would suggest going by Python version instead of package version personally, and it better reflects the NumPy support matrix and reduces the search space for what should be the same locked versions.