Cannot load pickled UMAP: No module named 'numba.core'

[WallBPG]

I first had issues building the docker file. When docker attempts to build using docker-compose build, conda was unable to use a flexible solve when running the following step in the Dockerfile:

[+] Building 291.1s (6/11)                                                         docker:desktop-linux
 => [ancestry internal] load build definition from Dockerfile                                      0.0s
 => => transferring dockerfile: 921B                                                               0.0s
 => [ancestry internal] load metadata for docker.io/continuumio/miniconda3:4.8.2                   0.3s
 => [ancestry internal] load .dockerignore                                                         0.0s
 => => transferring context: 2B                                                                    0.0s
 => CACHED [ancestry 1/7] FROM docker.io/continuumio/miniconda3:4.8.2@sha256:456e3196bf3ffb13fee7  0.0s
 => [ancestry internal] load build context                                                         0.0s
 => => transferring context: 61.80kB                                                               0.0s
 => ERROR [ancestry 2/7] RUN conda install -y       -c conda-forge       matplotlib=3.2.1 xgboo  290.7s
------    

RUN conda install -y \
      -c conda-forge \
      matplotlib=3.2.1 xgboost=1.0.2  \
      pandas=1.0.1 numpy=1.18.1 \
      scipy=1.4.1 umap-learn=0.4.2 \
      scikit-learn=0.24.1 \
      bokeh=2.4.3 && \
      # clean up
      conda clean --all

To get around not being able to build the image, I used the pre-built image here.

python3 -m igm_churchill_ancestry --help works as expected. When running a test with real data, the program has trouble loading the required .pkl files.

Cannot load pickled UMAP: No module named 'numba.core'

...

File ..., line 47, in transform_input
    embedding = umap.transform(xsvd)

The error appears to be a problem with various versions installed in my environment. Here's what I have installed:

> _libgcc_mutex             0.1                 conda_forge    conda-forge
> _openmp_mutex             4.5                       2_gnu    conda-forge
> awscli                    1.20.49                  pypi_0    pypi
> bokeh                     2.4.3                    pypi_0    pypi
> boto3                     1.18.49                  pypi_0    pypi
> botocore                  1.21.49                  pypi_0    pypi
> ca-certificates           2024.2.2             hbcca054_0    conda-forge
> colorama                  0.4.3                    pypi_0    pypi
> cycler                    0.12.1                   pypi_0    pypi
> docutils                  0.15.2                   pypi_0    pypi
> importlib-metadata        7.1.0                    pypi_0    pypi
> jinja2                    3.1.3                    pypi_0    pypi
> jmespath                  0.10.0                   pypi_0    pypi
> joblib                    1.3.2                    pypi_0    pypi
> kiwisolver                1.4.5                    pypi_0    pypi
> ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
> libffi                    3.3                  h58526e2_2    conda-forge
> libgcc-ng                 13.2.0               h807b86a_5    conda-forge
> libgomp                   13.2.0               h807b86a_5    conda-forge
> libsqlite                 3.45.2               h2797004_0    conda-forge
> libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
> libzlib                   1.2.13               hd590300_5    conda-forge
> llvmlite                  0.31.0                   pypi_0    pypi
> markupsafe                2.1.5                    pypi_0    pypi
> matplotlib                3.2.1                    pypi_0    pypi
> ncurses                   6.4.20240210         h59595ed_0    conda-forge
> numba                     0.48.0                   pypi_0    pypi
> numpy                     1.19.0                   pypi_0    pypi
> openssl                   1.1.1w               hd590300_0    conda-forge
> packaging                 24.0                     pypi_0    pypi
> pandas                    1.0.1                    pypi_0    pypi
> pillow                    10.2.0                   pypi_0    pypi
> pip                       24.0               pyhd8ed1ab_0    conda-forge
> pyasn1                    0.5.1                    pypi_0    pypi
> pynndescent               0.5.11                   pypi_0    pypi
> pyparsing                 3.1.2                    pypi_0    pypi
> python                    3.8.6           hffdb5ce_5_cpython    conda-forge
> python-dateutil           2.9.0.post0              pypi_0    pypi
> pytz                      2024.1                   pypi_0    pypi
> pyyaml                    5.4.1                    pypi_0    pypi
> readline                  8.2                  h8228510_1    conda-forge
> rsa                       4.7.2                    pypi_0    pypi
> s3transfer                0.5.2                    pypi_0    pypi
> scikit-learn              0.24.1                   pypi_0    pypi
> scipy                     1.4.1                    pypi_0    pypi
> setuptools                69.2.0             pyhd8ed1ab_0    conda-forge
> six                       1.16.0                   pypi_0    pypi
> sqlite                    3.45.2               h2c6b66d_0    conda-forge
> tbb                       2021.11.0                pypi_0    pypi
> threadpoolctl             3.4.0                    pypi_0    pypi
> tk                        8.6.13          noxft_h4845f30_101    conda-forge
> tornado                   6.4                      pypi_0    pypi
> tqdm                      4.66.2                   pypi_0    pypi
> typing-extensions         4.10.0                   pypi_0    pypi
> umap-learn                0.4.2                    pypi_0    pypi
> urllib3                   1.26.18                  pypi_0    pypi
> wheel                     0.43.0             pyhd8ed1ab_0    conda-forge
> xgboost                   1.0.2                    pypi_0    pypi
> xz                        5.2.6                h166bdaf_0    conda-forge
> zipp                      3.18.1                   pypi_0    pypi
> zlib                      1.2.13               hd590300_5    conda-forge

[andreirajkovic]

Hi @WallBPG where are you running python3 -m igm_churchill_ancestry --help from? Inside the docker?

[WallBPG]

I could not get the docker image to build using the standard commands:

cd snvstory
docker-compose build

So, I used the pre-built image I mentioned above.

I also attempted to create my own environment, etc.

The main issue seems to be loading the .pkl with a different environment than it was created in. But, creating the same environment seems to be impossible solve with conda. Also, using .pkl files is generally not recommended for security concerns. See more info here

[andreirajkovic]

If you pulled down the docker image could you try running this command?

docker run \
  -v /path/to/your/data:/data \
  -v /path/to/your/output:/output \
  -t "heritagehelix/snvstory:3.0.2" \
  --path /data/your_input_file.vcf.gz \
  --resource /data/resource_dir \
  --output-dir /output \
  --genome-ver 38 \
  --mode WES

[jsl-netizen]

HI just to say i have the same issue i cannot build the docker image i have this error too
[+] Building 291.1s (6/11) docker:desktop-linux
=> [ancestry internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 921B 0.0s
=> [ancestry internal] load metadata for docker.io/continuumio/miniconda3:4.8.2 0.3s
=> [ancestry internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [ancestry 1/7] FROM docker.io/continuumio/miniconda3:4.8.2@sha256:456e3196bf3ffb13fee7 0.0s
=> [ancestry internal] load build context 0.0s
=> => transferring context: 61.80kB 0.0s
=> ERROR [ancestry 2/7] RUN conda install -y -c conda-forge matplotlib=3.2.1 xgboo 290.7s