Improve dependency management in InVEST

[phargogh]

We've had a couple of issues just in the past few months where a package that InVEST depends on is built in some faulty way. Recent examples of this include:

• Specific versions of rtree include the libspatialindex DLL, but fail to handle the import path correctly. (Released on pypi.org)
• One of the Gohlke builds of shapely had a faulty build (we never did figure out exactly what went wrong).
• There was a GDAL issue on conda-forge, where one of the underlying C libraries was unvendored and this messed up the dependency chain and broke all our builds for a day or two.

The only way we could (sometimes) work around these issues was to explicitly forbid a specific package version in requirements.txt, which is like using a sledgehammer on a nail: functional, but crude. When a package is released on pypi.org, it's considered the single source of truth and it cannot be updated. This is not the case with Gohlke's packages, where he does sometimes rebuild a package and update it on his index. Only conda actually allows for specific builds to be available and referred to for each package version.

So, can we use conda to produce more reliable builds? Here's one process that might work:

1. We maintain a core set of requirements that represent the generic package versions needed for our software to function.
2. We also maintain a set of files, one for each target operating system and each target version of python we're testing against, that specify exactly what packages are to be installed into the conda environment for that python version on that OS.
3. Periodically (from time to time and also when we make changes to requirements.txt), we'll run a script to re-generate these derived files from requirements.txt. When these file changes go through the PR process, we'll verify that tests still work.

This pinning process would allow us to only update packages in our build/test environments when we actually need to and would also allow us to verify that the build process will work reliably with the packages specified.

[phargogh]

From within an activated conda environment, conda env export will generate a valid conda environment YML file that should recreate the environment completely, even including any packages that were installed via pip.

By way of example, this is the environment created via

$ conda create -p ./env37-conda-demo -c conda-forge python=3.7 flask
$ conda activate ./env37-conda-demo
(./env37-conda-demo) $ pip install pandas

And then the conda env export generated looks like this, which includes all of the dependencies recorded and channels used (including pip-installed packages):

name: /Users/jdouglass/workspace/phargogh/invest/env37-conda-demo
channels:
  - conda-forge
  - defaults
dependencies:
  - ca-certificates=2020.6.20=hecda079_0
  - certifi=2020.6.20=py37hc8dfbb8_0
  - click=7.1.2=pyh9f0ad1d_0
  - flask=1.1.2=pyh9f0ad1d_0
  - itsdangerous=1.1.0=py_0
  - jinja2=2.11.2=pyh9f0ad1d_0
  - libcxx=10.0.0=h1af66ff_2
  - libffi=3.2.1=h4a8c4bd_1007
  - markupsafe=1.1.1=py37h9bfed18_1
  - ncurses=6.1=h0a44026_1002
  - openssl=1.1.1g=h0b31af3_0
  - pip=20.1.1=py_1
  - python=3.7.6=cpython_h1fd5dd1_6
  - python_abi=3.7=1_cp37m
  - readline=8.0=hcfe32e1_0
  - setuptools=49.1.0=py37hc8dfbb8_0
  - sqlite=3.32.3=h93121df_0
  - tk=8.6.10=hbbe82c9_0
  - werkzeug=1.0.1=pyh9f0ad1d_0
  - wheel=0.34.2=py_1
  - xz=5.2.5=h0b31af3_0
  - zlib=1.2.11=h0b31af3_1006
  - pip:
    - numpy==1.19.0
    - pandas==1.0.5
    - python-dateutil==2.8.1
    - pytz==2020.1
    - six==1.15.0
prefix: /Users/jdouglass/workspace/phargogh/invest/env37-conda-demo

We should therefore be able to:

• generate one of these for each combination of {python version, operating system, architecture} we're testing against.
• use these generated files in our CI jobs for better control over dependencies
• include these generated files in our binary build distributions for easier debugging of our build environment (which could be useful for environment debugging when we have a broken binary build)
• hash these files in actions workflows to improve build runtime via package caching
• (later on) set up automation to attempt to regenerate these files with the latest versions of our dependencies.

Unresolved issues:

• requirements.txt has historically been the single source of truth for CI dependencies, and so we'll need to rethink the relationship between the two dependency managers. Can one be generated from the other?

[dcdenu4]

I can confirm that for 64-bit Windows builds conda works for testing as is implemented currently for the 64-bit Mac testing framework. I do NOT believe conda will work for 32-bit builds as there haven't been GDAL 32 bit binaries since like GDAL 2.2.4 in the gdal feedstock. We would have to get GDAL from Gohlke still to support 32 bit.

[phargogh]

Update

Now that we've had a little while to let this sit for a little while, I thought it might make sense to revisit some of the things we've learned about this.

Problem

Always using the very latest versions of packages means that we're vulnerable to having builds fail when there's an unexpected issue with an upstream package when the package is updated. It would be preferable to have our specific build dependencies tracked in such a way that our CI services can reliably build and test using specific package versions. This would allow us the opportunity to deliberately update packages when appropriate, and also would enable us to more easily determine what packages were used in a specific version of InVEST.

Considerations

1. How should dependencies be specified?
   How might we specify dependencies that's useful to our build and test workflows on CI services? Could this approach also be used in our own local development and testing?

2. Process for deliberately updating dependencies in a controlled manner.
   How might we go about updating the dependencies of InVEST in a controlled manner as libraries continue to evolve?

3. Balance between natcap.invest the library and InVEST the application
   Since InVEST is both an application and a library, how might we have general requirements specifications (appropriate for a library) and also track specific dependencies that are used in the application build? How might we manage duplication between the two?

Adding @dcdenu4 to this since he had expressed interest in looking into a solution. Did I miss anything from the above @dcdenu4 ?

[phargogh]

This issue came up again over in #635 (comment). @emlys has a good suggestion, specifically:

@phargogh, this makes me wonder if we should start setting an upper bound on every dependency, to the latest version that we've tested with at the time of a release? I know we've tried to be as unrestrictive as possible with dependencies. But with enough time, pip install natcap.invest==<old version> will always eventually break as dependencies get non-backwards-compatible updates. I was hoping pip install would have some option to install the minimum allowed version of each dependency instead of the maximum, but I'm not finding one.

Getting back to the age-old question of whether InVEST is a library or an application, I'm starting to lean more towards it being an application. And if that's the case, then we should be more prescriptive about what versions of packages can be installed.

Since part of the difficulty here is in the fact that our environment cannot be completely described by requirements.txt (GDAL's dependencies aren't managed by python, for example), maybe this is another reason to have an official docker release for each official InVEST release.

[davemfish]

I think some of the solutions described above could be accomplished with a lockfile approach, such as https://conda.github.io/conda-lock/

One recurring problem we have is that PyInstaller tends to be one step behind other libraries, since libraries do not ensure their own PyInstaller compatibility. So a typical pattern is,

• library (e.g. SciPy) does a release
• Our PyInstaller binaries break because of a new hidden import or something in scipy
• Issue is reported to PyInstaller and a fix comes in the next release

Basically, we get the latest SciPy immediately after it is released, but a compatible PyInstaller release comes weeks later.

If we used a lockfile, our CI would not install the latest available packages. But I could imagine some automation that periodically updates all packages and runs our full build and test workflows. Similar to our automated release process, it could do a PR with the updated lockfile. If all tests pass, we merge it, if they don't we address the compatibility issues.

I assume we can still have an environment.yml file with restrictions similar to what we have now. So a conda lock --update --file environment.yml would update all packages, but staying within the restrictions we declare.