Pass Install Extras to Markers #9553

reesehyde · 2024-07-15T17:24:43Z

Pull Request Check List

Resolves:

This PR exposes the extras supplied at install to the 'extra' marker for dependencies, taking up @radoering's offer to create a PR implementing this functionality to allow switching torch installation versions based on a cuda extra.

It is a duplicate of the feature in python-poetry/poetry-core#613 but it appears that PR is no longer active, with the last updates in August 2023 and an unanswered question about timeline from March 2024. This PR takes a different approach: I noticed that python-poetry/poetry-core#636 added special handling for the extras marker value but that marker isn't populated often, so I opted to populate the value when extras are specified in the installer.

See issues and unit tests for more details but the idea is to, among other things, enable exclusive extras like so:

[tool.poetry]
package-mode = false

[tool.poetry.dependencies]
python = "^3.10"
click = [
    { markers = "extra == 'one' and extra != 'two' and extra != 'three'", version = "8.1.1", optional = true},
    { markers = "extra != 'one' and extra == 'two' and extra != 'three'", version = "8.1.2", optional = true},
    { markers = "extra != 'one' and extra != 'two' and extra == 'three'", version = "8.1.3", optional = true}
 ]

[tool.poetry.extras]
one = ["click"]
two = ["click"]
three = ["click"]

@radoering it looks like you're very familiar with this issue, would you be the right person to review? 🙏
I'm sure I'm missing something here but look forward to iterating!

Edit by @radoering: fix links to poetry-core PRs

Added tests for changed code.
Updated documentation for changed code.

… with active root extras

radoering · 2024-07-16T15:45:58Z

I will try to take a closer look at the end of the week.

radoering

This looks really good. Nice work. 👍

It looks like some of the changes are not tested yet or may be redundant. See separate comments for details.

src/poetry/puzzle/provider.py

radoering · 2024-07-20T12:57:17Z

src/poetry/puzzle/provider.py

@@ -601,7 +620,9 @@ def complete_package(

            # For dependency resolution, markers of duplicate dependencies must be
            # mutually exclusive.
-            active_extras = None if package.is_root() else dependency.extras
+            active_extras = (
+                self._active_root_extras if package.is_root() else dependency.extras


Seems like that is not tested yet. (Tests succeed without this change.) You probably have to add a test similar to test_solver_resolves_duplicate_dependency_in_extra, which tests this for non root extras.

Ah yep, added test_solver_resolves_duplicate_dependency_in_root_extra(). But this results in both versions of A being selected, see the note about the bug here

radoering · 2024-07-20T13:05:07Z

src/poetry/puzzle/provider.py

@@ -541,7 +555,12 @@ def complete_package(
            if dep.name in self.UNSAFE_PACKAGES:
                continue

-            if self._env and not dep.marker.validate(self._env.marker_env):
+            active_extras = (
+                self._active_root_extras if package.is_root() else dependency.extras


Seems like the non root part is not tested yet. (Tests succeed when replacing dependency.extras with None.) Maybe, this path can be triggered by taking one of your tests and introducing an intermediate package?

Great catch! While trying to add a test for this I uncovered an issue referenced in a top-level comment. The breaking test I've added doesn't include the intermediate package for simplicity but I can add it as we get to the bottom of that.

radoering · 2024-07-20T13:10:10Z

src/poetry/puzzle/provider.py

+            and (
+                not self._env
+                or dep.marker.validate(
+                    self._marker_values(
+                        self._active_root_extras
+                        if dependency_package.package.is_root()
+                        else dependency_package.dependency.extras
+                    )
+                )
+            )


It looks like this change is not required by any test. To be honest, I have currently no idea if this change is redundant or if a test is missing...

Ah yeah I wasn't sure what the best policy was here: I just updated the places where I saw markers being evaluated to include extras. What I found is that of the three changes in provider.py, any one of them is sufficient to pass the first round of tests. So I think there is some redundancy, but my thought is to not eliminate anything yet due to the issue with uncovered around not resolving conflicts with their own different extras.

reesehyde · 2024-07-25T02:35:27Z

Thank you so much for the great feedback here! All the comments make sense to me, I'm working through it and will submit updates and responses later this week

GeeCastro · 2024-08-22T17:41:32Z

How would this differentiate between the extras for the project itself and the extras for a dependency package?

For the project the scenario would be "I want to install my project with cuda124 extra" which I guess you could interface with the -E/--extra option

poetry install -E cuda124

and define the in the package

[tool.poetry.extras]
cuda124 = ["torch"]

Here your suggestion would add

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    { markers = "extra == 'cuda124'", version = "2.4.0+cu124", optional = true},
                ^^^^^^

And a package-specific extra for example

poetry add "pandas[aws]"

which would show in the extras field as follows

[tool.poetry.dependencies]
python = "^3.11"
pandas = {extras = ["aws"], version = "^2.2.2"}
          ^^^^^

Would the differentiation be made on the following?

project extras in the marker string
package's extras in the extras field

If so, I must have misunderstood the docs because I thought these 2 syntaces express the same requirements/constraints

reesehyde · 2024-08-29T18:55:05Z

I apologize for the long delay here, I had to pause this before I managed to finish all the updates. But it hasn't left my mind, and I'll be able to resume the updates in a couple weeks and will push them up and respond to everything in mid-September! Thanks for the patience

mg515 · 2024-09-25T10:31:42Z

@reesehyde Any update on this? Would love to have this feature available. ^^

reesehyde

I apologize for the long delay here, I've been working on the suggested updates but uncovered an issue in the process. I've added a couple breaking tests which will hopefully help illuminate it. But I'm a little stuck on the best course of action, so would love a push in the right direction if you're able @radoering.

It seems that part of the reason the initial tests passed is because they used conflicting versions of torch, which works only because they then go through the process of merging duplicates since the package name is the same in all cases.

However, dependencies with their own differing extras (e.g. torch[cpu] and torch[cuda]) are not considered duplicates, so they result in a SolverProblemError.
Simply considering them duplicates creates its own issues (e.g. see the test_solver_resolves_duplicate_dependency_in_extra test). So we need some way to do overlapping marker resolution in these cases without breaking other solving cases.

Another possibility would be scoping this PR down to not support conflicting dependency extras, but I'm not certain how that would look.

reesehyde · 2024-10-01T20:26:38Z

src/poetry/puzzle/provider.py

@@ -601,7 +620,9 @@ def complete_package(

            # For dependency resolution, markers of duplicate dependencies must be
            # mutually exclusive.
-            active_extras = None if package.is_root() else dependency.extras
+            active_extras = (
+                self._active_root_extras if package.is_root() else dependency.extras


Ah yep, added test_solver_resolves_duplicate_dependency_in_root_extra(). But this results in both versions of A being selected, see the note about the bug here

src/poetry/puzzle/provider.py

reesehyde · 2024-10-11T20:30:25Z

src/poetry/puzzle/provider.py

+            and (
+                not self._env
+                or dep.marker.validate(
+                    self._marker_values(
+                        self._active_root_extras
+                        if dependency_package.package.is_root()
+                        else dependency_package.dependency.extras
+                    )
+                )
+            )


Ah yeah I wasn't sure what the best policy was here: I just updated the places where I saw markers being evaluated to include extras. What I found is that of the three changes in provider.py, any one of them is sufficient to pass the first round of tests. So I think there is some redundancy, but my thought is to not eliminate anything yet due to the issue with uncovered around not resolving conflicts with their own different extras.

reesehyde · 2024-10-11T20:33:32Z

src/poetry/puzzle/provider.py

@@ -541,7 +555,12 @@ def complete_package(
            if dep.name in self.UNSAFE_PACKAGES:
                continue

-            if self._env and not dep.marker.validate(self._env.marker_env):
+            active_extras = (
+                self._active_root_extras if package.is_root() else dependency.extras


Great catch! While trying to add a test for this I uncovered an issue referenced in a top-level comment. The breaking test I've added doesn't include the intermediate package for simplicity but I can add it as we get to the bottom of that.

reesehyde · 2024-10-11T20:56:13Z

How would this differentiate between the extras for the project itself and the extras for a dependency package?

@GeeCastro this is a great question, you're right that they are basically the same syntax! The problem comes up when you have conflicting extras, e.g. if the cuda extra requires torch == 1.0.0+cuda while the cpu extra requires torch == 1.0.0+cpu. Packages must be resolvable with every possible combination of extras, so if you tried to use both (--extra cpu --extra cuda) it would be impossible since 1.0.0+cuda conflicts with 1.0.0+cpu. Markers provide a way to say "cuda extra requires torch == 1.0.0+cuda, but only if the cpu extra isn't also present" so they don't conflict.

radoering · 2024-10-19T16:21:26Z

tests/puzzle/test_solver.py

+@pytest.mark.parametrize("with_extra", [False, True])
+def test_solver_resolves_duplicate_dependency_in_root_extra(
+    package: ProjectPackage,
+    pool: RepositoryPool,
+    repo: Repository,
+    io: NullIO,
+    with_extra: bool,
+) -> None:
+    """
+    Without extras, a newer version of A can be chosen than with root extras.
+    """


This is only a theoretical issue. In other words, the setup of the test cannot happen when invoking poetry.

First of all, you cannot create this issue with Poetry 1.8 because if you try something like:

[tool.poetry.dependencies] python = "^3.9" tomli = [ { version = ">=1" }, { version = "^1", optional = true }, ] [tool.poetry.extras] foo = [ "tomli" ]

... both dependencies are in the extra. However, with the main branch (by the way, please rebase your branch when it suits you) and PEP 621 support you can define:

[project] # ... requires-python = ">=3.9" dependencies = [ "tomli>=1", ] [project.optional-dependencies] foo = [ "tomli>=1,<2", ]

When you debug this, you will notice that the tomli dependency from the extra has no extra == "foo" marker. Generally speaking, dependencies of root extras do not have the extra in the marker but only the attribute _in_extras set. Thus, a more realistic test will look like that (and does not fail):

def test_solver_resolves_duplicate_dependency_in_root_extra( package: ProjectPackage, pool: RepositoryPool, repo: Repository, io: NullIO, ) -> None: """ Without extras, a newer version of A could be chosen than with root extras. However, only the older version is chosen. """ package_a1 = get_package("A", "1.0") package_a2 = get_package("A", "2.0") dep = get_dependency("A", ">=1.0") package.add_dependency(dep) dep_extra = get_dependency("A", "^1.0", optional=True) dep_extra._in_extras = ["foo"] package.extras = {canonicalize_name("foo"): [dep_extra]} package.add_dependency(dep_extra) repo.add_package(package_a1) repo.add_package(package_a2) solver = Solver(package, pool, [], [], io) transaction = solver.solve() check_solver_result(transaction, ([{"job": "install", "package": package_a1}]))

I only checked this test for now and will take a look at the installer tests later.

reesehyde added 13 commits July 13, 2024 20:19

add breaking unit test for conflicting pytorch cpu/cuda extras

c35676f

support project extras in markers: populate 'extras' marker including…

664daa8

… with active root extras

add locking test

900913c

fix default when active root extras not passed

1ee3a90

add unit tests for python-poetry#834

6b55283

consolidate tests

57ed6f7

add documentation

dc86259

fix types

afdab51

fix formatting

6b3f24e

simplify exclusive extras example

de6e032

run ruff formatter

ec5592f

fix key not in dict format

4de59bd

fix pre-commit and full run

203c037

Secrus requested a review from radoering July 16, 2024 09:22

radoering requested changes Jul 20, 2024

View reviewed changes

reesehyde added 3 commits October 11, 2024 15:37

add markers assert and remove unnecessary check

c95e3cf

add test for root extras

8490954

add broken tests for conflicting dependency extras

fd1c1dc

reesehyde commented Oct 11, 2024

View reviewed changes

radoering reviewed Oct 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pass Install Extras to Markers #9553

Pass Install Extras to Markers #9553

reesehyde commented Jul 15, 2024 •

edited by radoering

Loading

radoering commented Jul 16, 2024 •

edited

Loading

radoering left a comment

radoering Jul 20, 2024

reesehyde Oct 1, 2024

radoering Jul 20, 2024

reesehyde Oct 11, 2024

radoering Jul 20, 2024

reesehyde Oct 11, 2024

reesehyde commented Jul 25, 2024

GeeCastro commented Aug 22, 2024 •

edited

Loading

reesehyde commented Aug 29, 2024

mg515 commented Sep 25, 2024

reesehyde left a comment

reesehyde Oct 1, 2024

reesehyde Oct 11, 2024

reesehyde Oct 11, 2024

reesehyde commented Oct 11, 2024

radoering Oct 19, 2024

Pass Install Extras to Markers #9553

Are you sure you want to change the base?

Pass Install Extras to Markers #9553

Conversation

reesehyde commented Jul 15, 2024 • edited by radoering Loading

Pull Request Check List

radoering commented Jul 16, 2024 • edited Loading

radoering left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reesehyde commented Jul 25, 2024

GeeCastro commented Aug 22, 2024 • edited Loading

reesehyde commented Aug 29, 2024

mg515 commented Sep 25, 2024

reesehyde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reesehyde commented Oct 11, 2024

Choose a reason for hiding this comment

reesehyde commented Jul 15, 2024 •

edited by radoering

Loading

radoering commented Jul 16, 2024 •

edited

Loading

GeeCastro commented Aug 22, 2024 •

edited

Loading