Process wheelhouse.txt holistically rather than per-layer #569
Conversation
Processing the wheelhouse.txt file from each layer in isolation means that pip can't resolve conflicts, duplicates, or dependency version pinning which spans across layers. This can lead to duplicate versions of libraries being installed which violate constraints to avoid certain upstream bugs. This came up in both of: * https://bugs.launchpad.net/charm-flannel/+bug/1884598 * https://bugs.launchpad.net/charm-octavia/+bug/1884164 And in both cases trying to resolve it on a per-layer or per-charm basis results in a whack-a-mole process of catching all of the layers which indirectly reintroduce the unpinned version of a dependency pinned in a base layer.
This allows higher layers to override explicit requirements while still inheriting specs from lower layers.
johnsca
changed the title
There was a problem hiding this comment.
Tested this with Octavia. I am still getting two versions of netaddr:
netaddr-0.7.19.tar.gz netaddr-0.7.20.tar.gz
0.7.19 we want and is pinned in layer basic. I suspect neutron in the Octavia wheelhoust.txt is pulling in the newer version as many many packages depend on it and may or may not have constraints.
| help="Show post-build report of changes") | ||
| parser.add_argument('-R', '--no-report', action="store_false", |
There was a problem hiding this comment.
-r and -R seem to be contradictory; how about we kill the -R codepath and just use -r=True|False instead?
There was a problem hiding this comment.
I tried to figure out a way to implement the -r[=true/|false] pattern but the case that always breaks is the current behavior that many may be using:
$ charm build -r ./src
There's no way with argparse to say "consume an arg, but only if it matches one of these choices". It will either not consume any args, always consume an arg, or try to consume an arg but blow up if it doesn't match one of the choices.
Additionally, I found at least one example with git -p and git -P which have the same contradictory meaning, and I know I've seen other cases and not found it terribly confusing. In general, and in the case of this change, the behavior is what I would expect: either of the options can be given, even multiple times, and the last one present on the command line takes precedence.
There was a problem hiding this comment.
@johnsca i appreciate the attempt. I didn't intend for you to bend over backwards, but it sure would have been nice if "honor a bool value" would just work for argparse optargs. Anywho, I'm annoyed but not blocking contradictory flags; as you said, last in gets the cake, and i think that's a reasonable UX that charm builders can shoulder.
All that said, given that --report is now the default, I would ask you to consider #570. That is, deprecate -r with some stderr message that says -r is default and therefore deprecated; use -R to prevent charm build reports.
|
This has been released in the snap and to PyPI with 2.7.5 |
Processing the wheelhouse.txt file from each layer in isolation means that pip can't resolve conflicts, duplicates, or dependency version pinning which spans across layers. This can lead to duplicate versions of libraries being installed which violate constraints to avoid certain upstream bugs.
This came up in both of:
And in both cases trying to resolve it on a per-layer or per-charm basis results in a whack-a-mole process of catching all of the layers which indirectly reintroduce the unpinned version of a
dependency pinned in a base layer.