Fix build and tests; update Python versions / dependencies #569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

fedarko wants to merge 20 commits into biocore:master from fedarko:fix-tests-202504

.github/workflows/main.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -19,9 +19,16 @@ jobs: @@
         # The type of runner that the job will run on (available options are window/macOS/linux)
         runs-on: ubuntu-latest
-        # used in McHelper (similar to TRAVIS_PULL_REQUEST variable)
         env:
+          # used in McHelper (similar to TRAVIS_PULL_REQUEST variable)
           BRANCH_NUMBER: ${{ github.event.number }}
+          # currently, puppeteer gives us the following error:
+          # Error: Failed to launch the browser process!
+          # [4551:4551:0501/052725.614275:FATAL:zygote_host_impl_linux.cc(132)] No usable sandbox! If you are running on Ubuntu 23.10+ or another Linux distro that has disabled unprivileged user namespaces with AppArmor, see https://chromium.googlesource.com/chromium/src/+/main/docs/security/apparmor-userns-restrictions.md. Otherwise see https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md for more information on developing with the (older) SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
+          #
+          # Per the chromium.googlesource.com link above, it looks like setting
+          # this env variable can fix the issue?
+          CHROME_DEVEL_SANDBOX: /opt/google/chrome/chrome-sandbox
         # Steps represent a sequence of tasks that will be executed as part of the job
         steps:
@@ Expand All / @@ -32,16 +39,16 @@ jobs: @@
               fetch-depth: 0
           - name: Set up Node.js
-            uses: actions/setup-node@v1
-            with:
-              # Can use a matrix of Node.js versions in the future if desired;
-              # however, since Node.js should only be used when developing
-              # EMPress, it's not an urgent priority (and should likely be done
-              # separately from the Python CI stuff to save time)
-              node-version: 14
+            uses: actions/setup-node@v4
           - name: Install Node.js modules
-            run: npm install -g qunit-puppeteer jshint [email protected]
+            run: npm install -g node-qunit-puppeteer jshint [email protected]
+          - name: Set up chrome-sandbox permissions for Puppeteer
+            run: |
+              # from https://github.com/Chia-Network/chia-blockchain/issues/4721#issuecomment-841754056
+              sudo chown root:root $CHROME_DEVEL_SANDBOX
+              sudo chmod 4755 $CHROME_DEVEL_SANDBOX
           - name: Install flake8
             run: conda run -n base pip install flake8
@@ Expand All / @@ -50,22 +57,25 @@ jobs: @@
             run: |
               conda run -n base make stylecheck
-          - name: Set up conda
-            uses: s-weigand/setup-conda@v1
-            with:
-              update-conda: true
-              python-version: 3.6
-              conda-channels: anaconda, conda-forge
-          - name: Create QIIME 2 conda environment
+          - name: Download QIIME 2 conda environment YAML
             run: |
-              wget https://data.qiime2.org/distro/core/qiime2-2020.6-py36-linux-conda.yml
-              conda env create -n qiime2-dev --file qiime2-2020.6-py36-linux-conda.yml
+              wget -O q2-env.yml https://raw.githubusercontent.com/qiime2/distributions/dev/2024.10/amplicon/released/qiime2-amplicon-ubuntu-latest-conda.yml
+          # https://github.com/conda-incubator/setup-miniconda#example-3-other-options
+          # (from Qurro's qiime2.yml workflow)
+          - name: Install this QIIME 2 version
+            uses: conda-incubator/setup-miniconda@v3
+            with:
+              activate-environment: qiime2-dev
+              environment-file: q2-env.yml
           - name: Install EMPress
             run: |
               conda run -n qiime2-dev pip uninstall emperor --yes
               conda run -n qiime2-dev pip install -e .[all] --verbose
+              # to prevent the "linear-tree" tests from failing, require an iow vsn
+              # of at least 1.0.8. see https://github.com/biocore/empress/pull/555.
+              conda run -n qiime2-dev pip install "iow >= 1.0.8"
               conda run -n qiime2-dev qiime dev refresh-cache
           - name: Run tests
@@ Expand Down @@

.github/workflows/standalone.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -20,12 +20,24 @@ jobs:
  
        strategy:

          matrix:

            # based roughly on https://github.com/conda-incubator/setup-miniconda#example-1-basic-usage

            python-version: ["3.6", "3.7", "3.8", "3.9"]

            python-version: ["3.9", "3.10", "3.11", "3.12"]

        # used in McHelper (similar to TRAVIS_PULL_REQUEST variable)

        env:

          BRANCH_NUMBER: ${{ github.event.number }}

        # https://github.com/conda-incubator/setup-miniconda#use-a-default-shell

        # Use the same shell for all commands here, preserving the "context" of the

        # conda environment we set up. (This is a lot nicer-looking than prefixing

        # all of these commands with "conda run -n empress" or something.)

        #

        # See https://stackoverflow.com/q/69070754 and 'bash -c "help set"' for details

        # about what the "-el {0}" means here.

        # ^^ copied from metaLJA CI

        defaults:

          run:

            shell: bash -el {0}

        # Steps represent a sequence of tasks that will be executed as part of the job

        steps:

          # first grab branch from github

    @@ -35,28 +47,22 @@ jobs:
  
              fetch-depth: 0

          # https://github.com/conda-incubator/setup-miniconda#example-1-basic-usage

          - uses: conda-incubator/setup-miniconda@v2

          - uses: conda-incubator/setup-miniconda@v3

            with:

              activate-environment: empress

              python-version: ${{ matrix.python-version }}

          - name: Install conda packages

            shell: bash -l {0}

            run: conda install flake8 nose

          # Pinning numpy < 2 per https://github.com/biocore/improved-octo-waddle/issues/59

          - name: Install Cython & NumPy

            shell: bash -l {0}

            run: pip install cython "numpy >= 1.12.0"

            run: pip install cython "numpy >= 1.12.0,<2"

          - name: Install EMPress

            shell: bash -l {0}

            run: pip install -e .[all]

          # tests that don't import QIIME 2 dependencies

          - name: Run (non-QIIME 2) Python tests

            shell: bash -l {0}

            run: >

              nosetests

              pytest

              ./tests/python/test_cli.py

              ./tests/python/test_compression_utils.py

              ./tests/python/test_core.py

CONTRIBUTING.md

-Original file line number
+Diff line change
@@ Expand Up @@
     JavaScript code.
     ```bash
-    npm install -g qunit-puppeteer jshint [email protected]
+    npm install -g node-qunit-puppeteer jshint [email protected]
     ```
     If you don't have `npm` installed, you will need to install that first.
     **Note**: if you can't install puppeteer, the (JavaScript) test suite can be
     run using a web browser by opening the page in `tests/index.html`.
+    **Note**: Previously, we used [`qunit-puppeteer`](https://www.npmjs.com/package/qunit-puppeteer),
+    but at some point this stopped working -- running it from the terminal halted forever. Using
+    the more actively maintained [`node-qunit-puppeteer`](https://www.npmjs.com/package/node-qunit-puppeteer)
+    package fixes this issue -- so, although these packages have very similar names,
+    please make sure you are using the correct one.
     ## Running tests
     The Makefile contained in the root of the Empress repository is the easiest way
@@ Expand Down @@

Makefile

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -17,11 +17,13 @@ CSSLOCS = empress/support_files/css/*.css
  
    test: pytest jstest

    pytest:

    	nosetests tests/python

    	python3 -B -m pytest tests/python --cov empress

    jstest:

    	@# Note: this assumes you're running this on a Linux/macOS system

    	qunit-puppeteer file://$(shell pwd)/tests/index.html

    	@# Also: previously, we used "qunit-puppeteer", but that stopped

    	@# working at some point. "node-qunit-puppeteer" is maintained and works

    	node-qunit-puppeteer tests/index.html

    # Lints and checks code style

    stylecheck:

README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -43,12 +43,12 @@ post a question on the [QIIME 2 Forum](https://forum.qiime2.org)!)
  
    ### Standalone Version

    Empress is available through [PyPI](https://PyPI.org/project/empress/). We recommend installing Empress into an environment (e.g. a [conda](https://docs.conda.io/) environment) using a Python version of at least 3.6.

    Empress is available through [PyPI](https://PyPI.org/project/empress/). As of writing, Empress supports Python versions 3.9, 3.10, 3.11, and 3.12.

    Run the following commands to install Empress:

    ```bash

    pip install cython "numpy >= 1.12.0"

    pip install cython "numpy >= 1.12.0,<2"

    pip install empress

    ```

empress/tools.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -286,7 +286,13 @@ def match_inputs( @@
             if ignore_missing_samples:
                 # Works similarly to how Emperor does this: see
                 # https://github.com/biocore/emperor/blob/659b62a9f02a6423b6258c814d0e83dbfd05220e/emperor/core.py#L350
-                samples_without_metadata = table_samples - sm_samples
+                #
+                # NOTE: as of April 2025, we can't use a set as the index of a
+                # DataFrame. Since we don't care about the order of samples in
+                # the DataFrame, we can just get around this by converting the
+                # set of (table_samples - sm_samples) to a list, per
+                # https://stackoverflow.com/a/73778792
+                samples_without_metadata = list(table_samples - sm_samples)
                 padded_metadata = pd.DataFrame(
                     index=samples_without_metadata,
                     columns=sample_metadata.columns,
@@ Expand Down @@

setup.py

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -14,6 +14,8 @@
  
    __email__ = "[email protected]"

    # based on the text found in github.com/qiime/pynast

    # NOTE: these supported python versions (3.9 through 3.12) match iow 1.0.8;

    # as of writing, it does not yet support python 3.13.

    classes = """

        Development Status :: 5 - Production/Stable

        License :: OSI Approved :: BSD License

    @@ -23,10 +25,10 @@
  
        Topic :: Software Development :: User Interfaces

        Programming Language :: Python

        Programming Language :: Python :: 3 :: Only

        Programming Language :: Python :: 3.6

        Programming Language :: Python :: 3.7

        Programming Language :: Python :: 3.8

        Programming Language :: Python :: 3.9

        Programming Language :: Python :: 3.10

        Programming Language :: Python :: 3.11

        Programming Language :: Python :: 3.12

        Programming Language :: Python :: Implementation :: CPython

        Operating System :: OS Independent

        Operating System :: POSIX

    @@ -38,13 +40,14 @@
  
    with open('README.md') as f:

        long_description = f.read()

    # NOTE: Pinning iow due to issue described in

    # https://github.com/biocore/improved-octo-waddle/pull/48 and

    # https://github.com/biocore/empress/pull/555

    # NOTE: As described in https://github.com/biocore/empress/pull/555, iow needs

    # to be at least 0.1.3 for ordinary use of Empress. For testing, the "linear"

    # trees we check will cause versions of iow > 0.1.3 and < 1.0.8 to crash --

    # however, this should not impact ordinary use of Empress.

    base = ["numpy", "scipy", "pandas", "click",

            "jinja2", "scikit-bio", "biom-format", "iow==0.1.3",

            "jinja2", "scikit-bio", "biom-format", "iow>=0.1.3",

            "emperor>=1.0.2"]

    test = ["flake8", "nose"]

    test = ["flake8", "pytest", "pytest-cov"]

    all_deps = base + test

    setup(

    @@ -75,7 +78,7 @@
  
        # Yanked from Qurro: NumPy and Cython need to be installed before

        # trying to install EMPress / other pip packages.

        # https://github.com/biocore/qurro/blob/master/setup.py

        setup_requires=["cython", "numpy >= 1.12.0"],

        setup_requires=["cython", "numpy >= 1.12.0, < 2"],

        install_requires=base,

        extras_require={'test': test, 'all': all_deps},

        long_description=long_description,

tests/python/test_core.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -13,7 +13,7 @@ @@
     import skbio
     from skbio.util import assert_ordination_results_equal
-    from pandas.util.testing import assert_frame_equal
+    from pandas.testing import assert_frame_equal
     from os.path import exists
     from shutil import rmtree
     import biom
@@ Expand Down Expand Up / @@ -183,7 +183,12 @@ def test_init_tree_plot_extra_fm(self): @@
                 },
                 index=["weshould", "befiltered"]
             )
-            smooshed_fm = self.feature_metadata.append(extra_fm)
+            # Previously, we used self.feature_metadata.append() to do this
+            # (https://pandas.pydata.org/pandas-docs/version/1.4/reference/api/pandas.DataFrame.append.html)
+            # but this has since been removed from pandas. pd.concat() works as
+            # a replacement -- we are just adding the rows of extra_fm on to the
+            # end of self.feature_metadata.
+            smooshed_fm = pd.concat([self.feature_metadata, extra_fm])
             viz = Empress(self.tree, feature_metadata=smooshed_fm)
             self.assertFalse(viz.is_community_plot)
             assert_frame_equal(viz.tip_md, self.feature_metadata.loc[["a"]])
@@ Expand Down @@

tests/python/test_taxonomy_utils.py

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -57,9 +57,9 @@ def setUp(self):
  
            )

        def _check_basic_case_worked(self, split_fm, taxcols):

            """Checks that a given DataFrame (and list of split-up taxonomy columns)

               matches the expected output from running split_taxonomy() on

               self.feature_metadata.

            """Checks that a given DataFrame (and list of split-up taxonomy

               columns) matches the expected output from running split_taxonomy()

               on self.feature_metadata.

            """

            # Let's verify that split_fm looks how we expect it to look.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix build and tests; update Python versions / dependencies #569

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Fix build and tests; update Python versions / dependencies #569

Are you sure you want to change the base?

Uh oh!

Fix build and tests; update Python versions / dependencies #569

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!