setup: upgrading to adage==0.10.2 breaks yadage workflows
#220
Comments
|
@matthewfeickert @lukasheinrich maybe you have some ideas on what's going on here? |
- pin `adage==0.10.1`. `adage==0.10.2` forces upgrading to `networkx>=2.4` which makes our yadage workflows fail on the workflow engine side. https://github.com/yadage/adage/releases/tag/v0.10.2 reanahub#220
From the job logs it looks like you're running
The reason for setting lower bounds on
for more details. Though if you're running Python 2, I'm not sure how you ever installed |
|
It also looks like your reana-workflow-engine-yadage/setup.py Lines 43 to 55 in ac194f6 so I'm not sure how |
|
I'm not running Python 2, that's just the container where the job is being run. The Yadage workflow engine uses Python 3.8. |
👍 and I see that you're using to produce a requirements lockfile that has So unless you're also updating your reana-workflow-engine-yadage/setup.py Line 47 in ac194f6 there's never going to be a way for you to have Example:$ docker run --rm -ti python:3.9 /bin/bash
root@cb127e75ef9f:/# python -m venv venv && . venv/bin/activate
(venv) root@cb127e75ef9f:/# python -m pip install --upgrade 'pip<22.0' wheel pip-tools # at time of example bug with pip and pip-tools
root@cb127e75ef9f:/# printf "adage>=0.10.2\nnetworkx==1.11\n" > requirements.txt
root@cb127e75ef9f:/# cat requirements.txt
adage>=0.10.2
networkx==1.11
(venv) root@cb127e75ef9f:/# python -m piptools compile --generate-hashes --output-file requirements.lock requirements.txt
Could not find a version that matches networkx==1.11,>=2.4 (from -r requirements.txt (line 2))
Tried: 0.34, 0.34, 0.35, 0.35, 0.35.1, 0.35.1, 0.36, 0.36, 0.37, 0.37, 0.99, 0.99, 1.0, 1.0, 1.0.1, 1.0.1, 1.1, 1.1, 1.2, 1.2, 1.3, 1.3, 1.4, 1.4, 1.5, 1.5, 1.6, 1.6, 1.7, 1.7, 1.8, 1.8, 1.8.1, 1.8.1, 1.9, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.1, 1.10, 1.10, 1.11, 1.11, 1.11, 2.0, 2.1, 2.2, 2.3, 2.4, 2.4, 2.5, 2.5, 2.5.1, 2.5.1, 2.6, 2.6, 2.6.1, 2.6.1, 2.6.2, 2.6.2, 2.6.3, 2.6.3
Skipped pre-versions: 1.0rc1, 1.0rc1, 1.2rc1, 1.2rc1, 1.3rc1, 1.3rc1, 1.4rc1, 1.4rc1, 1.5rc1, 1.5rc1, 1.6rc1, 1.6rc1, 1.7rc1, 1.7rc1, 1.8rc1, 1.8rc1, 1.9rc1, 1.9rc1, 1.9rc1, 1.10rc2, 1.10rc2, 1.10rc2, 1.11rc1, 1.11rc1, 1.11rc2, 1.11rc2, 1.11rc2, 2.2rc1, 2.3rc3, 2.3rc4, 2.4rc1, 2.4rc2, 2.4rc2, 2.5rc1, 2.5rc1, 2.6rc1, 2.6rc1, 2.6rc2, 2.6rc2
There are incompatible versions in the resolved dependencies:
networkx==1.11 (from -r requirements.txt (line 2))
networkx>=2.4 (from adage==0.10.2->-r requirements.txt (line 1))
root@cb127e75ef9f:/# printf "adage>=0.10.2\nnetworkx\n" > requirements.txt
root@cb127e75ef9f:/# cat requirements.txt
adage>=0.10.2
networkx
(venv) root@cb127e75ef9f:/# python -m piptools compile --generate-hashes --output-file requirements.lock requirements.txt
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
# pip-compile --generate-hashes --output-file=requirements.lock requirements.txt
#
adage==0.10.2 \
--hash=sha256:9327e256226070ccf69477e7689ac81a3c1a792eadad5ba45749a6a46fdfdac8 \
--hash=sha256:f03a80773f1267463f9054cca8e53108b1eefe2f9056fba643a0070d41e429a4
# via -r requirements.txt
networkx==2.6.3 \
--hash=sha256:80b6b89c77d1dfb64a4c7854981b60aeea6360ac02c6d4e4913319e0a313abef \
--hash=sha256:c0946ed31d71f1b732b5aaa6da5a0388a345019af232ce2f49c766e2d6795c51
# via
# -r requirements.txt
# adageAm I missing something here about what you're doing? Or are you saying that you're already removing the pins and then letting |
Exactly this. Sorry for the misunderstanding. |
Okay cool. 👍 Can you give me a branch that has a minimal failing example? I know that you've already explained that apparently removing the |
(Thanks to @lukasheinrich for pointing out that I missed this.) @mvidalgarcia Sorry can you also try installing My guess is from reana-workflow-engine-yadage/setup.py Lines 49 to 50 in ac194f6 this is because of us (or at least the edit: I'm trying with $ git diff origin/master -- setup.py
diff --git a/setup.py b/setup.py
index 0fce759..c6b6b3a 100644
--- a/setup.py
+++ b/setup.py
@@ -44,9 +44,7 @@ install_requires = [
"checksumdir>=1.1.4,<1.2",
"graphviz>=0.12", # FIXME needed only if yadage visuale=True.
"mock>=3.0,<4",
- "networkx==1.11",
- "packtivity==0.14.21",
- "pydot2>=1.0.33", # FIXME needed only if yadage visuale=True.
+ "pydot>=1.0.29", # FIXME needed only if yadage visuale=True.
"pydotplus>=2.0.2", # FIXME needed only if yadage visuale=True.
"pygraphviz>=1.5", # FIXME needed only if yadage visuale=True.
"reana-commons[yadage]>=0.8.0,<0.9.0",at the moment, and then updating the $ git diff origin/master -- requirements.txt | grep '==' | grep '+' | grep 'adage\|packtivity\|pydot'
+adage==0.10.2
+packtivity==0.14.24
+pydot==1.4.2
+pydotplus==2.0.2
+reana-commons[yadage]==0.8.2
+yadage==0.20.1
+yadage-schemas==0.10.7which I can get to build fine with ./run-tests.sh --check-docker-buildbut I'm not clear on how to run your minimal failing example. |
|
@matthewfeickert Your diff did the trick! Indeed it seems the error was caused by Thanks for the quick reaction! Edit: Here are the changes: https://github.com/reanahub/reana-workflow-engine-yadage/compare/4b6e2bd8b37566ffef132291fb32ef796a3c3a7b..25d17851d37ec285c9103a15b1cf3b8c0c856623 |
@mvidalgarcia Great! :) Yeah, sorry about all of this as this is indeed the fault of recent yadage org stuff as we've been trying to clean things up. Specifically, when going to On the upside though, I've made an |
|
The regression seen in #219 is caused by yadage/yadage#116. |
After upgrading to
adage==0.10.2, it's required to upgradenetworkx>=2.4. This makes our yadage workflows fail on the workflow engine side. The jobs finished successfully but there's a problem afterward, perhaps when the visualization files are generated.With
adage==0.10.1(andnetworkx==1.11) everything runs as usual.Adage release: https://github.com/yadage/adage/releases/tag/v0.10.2
The text was updated successfully, but these errors were encountered: