No module named 'diffusers.pipelines.controlnet_xs'

[Winnie202]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (main) or latest version (0.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmagic

Environment

Python: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 2147483648
GPU 0: NVIDIA TITAN X (Pascal)
CUDA_HOME: /usr/local/cuda-10.2
NVCC: Cuda compilation tools, release 10.2, V10.2.8
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.12.1+cu102
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 10.2
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
• CuDNN 7.6.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.13.1+cu102
OpenCV: 4.8.1
MMEngine: 0.8.4
MMCV: 2.0.1
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.2
MMagic: 1.2.0dev0+1bdd952

Reproduces the problem - code sample

from mmagic.apis import MMagicInferencer

Reproduces the problem - command or script

python demo/mmagic_inference_demo.py

Reproduces the problem - error message

ie_xiong/SR2/mmagic/yfolder/mmagic_inference_demo.py
/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
Traceback (most recent call last):
File "/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/diffusers/utils/import_utils.py", line 710, in _get_module
return importlib.import_module("." + module_name, self.name)
File "/media/veily/work/envs/openmmlab/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'diffusers.pipelines.controlnet_xs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/media/veily/other/winnie_xiong/SR2/mmagic/yfolder/mmagic_inference_demo.py", line 5, in
from mmagic.apis import MMagicInferencer
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/apis/init.py", line 2, in
from .inferencers.inference_functions import init_model
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/apis/inferencers/init.py", line 16, in
from .translation_inferencer import TranslationInferencer
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/apis/inferencers/translation_inferencer.py", line 12, in
from mmagic.models.base_models import BaseTranslationModel
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/models/init.py", line 6, in
from .editors import * # noqa: F401, F403
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/models/editors/init.py", line 2, in
from .animatediff import AnimateDiff, UNet3DConditionMotionModel
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/models/editors/animatediff/init.py", line 2, in
from .animatediff import AnimateDiff
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/models/editors/animatediff/animatediff.py", line 15, in
from mmagic.models.archs import TokenizerWrapper, set_lora
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/models/archs/init.py", line 80, in
register_diffusers_models()
File "/media/veily/other/winnie_xiong/SR2/mmagic/mmagic/models/archs/init.py", line 69, in register_diffusers_models
pipeline = getattr(diffusers.pipelines, pipeline_name)
File "/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/diffusers/utils/import_utils.py", line 698, in getattr
value = self._get_module(name)
File "/media/veily/work/envs/openmmlab/lib/python3.8/site-packages/diffusers/utils/import_utils.py", line 712, in _get_module
raise RuntimeError(
RuntimeError: Failed to import diffusers.pipelines.controlnet_xs because of the following error (look up to see its traceback):
No module named 'diffusers.pipelines.controlnet_xs'

Additional information

No response

[ghost]

Hello, I also encountered this problem just now, I reduced the diffusers version to 0.24.0 and this error did not occur.

[ghost]

pip install diffusers==0.24.0

[carlomigliorini]

I had the same problem, I installed diffusers==0.24.0 and the problem has been solved, but now I get a new error:
ModuleNotFoundError: No module named 'mmcv._ext'
The mmcv version is 2.0.1.
Any idea please ?

[leozjr]

followed by this offical guide.
I had the same problem, installed diffusers==0.24.0 not work.

/home/leo/.conda/envs/leo_inpainting/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/leo/.conda/envs/leo_inpainting/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/leo/.conda/envs/leo_inpainting/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/leo/.conda/envs/leo_inpainting/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/leo/.conda/envs/leo_inpainting/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
Traceback (most recent call last):
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/test.py", line 2, in <module>
    sd_inferencer = MMagicInferencer(model_name='stable_diffusion')
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/apis/mmagic_inferencer.py", line 147, in __init__
    self.inferencer = Inferencers(
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/apis/inferencers/__init__.py", line 81, in __init__
    self.inferencer = Text2ImageInferencer(
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/apis/inferencers/base_mmagic_inferencer.py", line 59, in __init__
    register_all_modules()
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/utils/setup_env.py", line 25, in register_all_modules
    import mmagic.evaluation  # noqa: F401,F403
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/evaluation/__init__.py", line 2, in <module>
    from .evaluator import Evaluator
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/evaluation/evaluator.py", line 12, in <module>
    from .metrics.base_gen_metric import GenMetric
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/evaluation/metrics/__init__.py", line 12, in <module>
    from .niqe import NIQE, niqe
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/evaluation/metrics/niqe.py", line 12, in <module>
    from mmagic.datasets.transforms import MATLABLikeResize
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/datasets/transforms/__init__.py", line 2, in <module>
    from .albu_function import AlbuCorruptFunction, PairedAlbuTransForms
  File "/home/leo/MentorWorks/ImageInpainting/MMLab/mmagic/mmagic/datasets/transforms/albu_function.py", line 4, in <module>
    import albumentations as albu
ModuleNotFoundError: No module named 'albumentations'

after pip install albumentations:

raise EnvironmentError(
OSError: runwayml/stable-diffusion-v1-5 does not appear to have a file named config.json.

[umaximu]

i'm having the same issue! i'm on a macbook air (m1), have followed several different official guides, and am about to try changing my python model with pyenv. which version of python have y'all been running?

[lewishaas]

I think this is an issue with the diffusers library. I commented out the line referring to controlnet_xs in site-packages/diffusers/pipelines/__init__.py and it worked afterward. I didn't change the diffusers version

[hullout]

pip install diffusers==0.24.0

it works, thanks

[mcsmart76]

I have not been able to resolve the issue with diffusers=0.24.0 nor by commenting out the controlnet_xs line.

I think this is an issue with the diffusers library. I commented out the line referring to controlnet_xs in site-packages/diffusers/pipelines/__init__.py and it worked afterward. I didn't change the diffusers version

This got me to a new error message: ModuleNotFoundError: No module named 'mmcv._ext'

My full repro from head using virtual Python environments:

$ cat ~/mmagic.sh 
#!/bin/bash

VENV=~/.mmagic

# To cleanup, run `deactivate` and `rm -r $VENV`.

# Create my virtual environment at $VENV and activate it.
python3 -m venv $VENV
source $VENV/bin/activate

# Dependency installs, trying to be as up to date as possible.
pip3 install 'torch>=2.3.0'
pip3 install torchvision
pip3 install -U openmim
# mmcv was bumped to 2.2.0 on 2024-04-16, which is incompatible
# with mmengine's version check as of 2024-04-25.
# https://github.com/open-mmlab/mmcv/pull/3087/commits
mim install 'mmcv>=2.0.0,<=2.1.0'
mim install mmengine
mim install mmagic

# Validate imports.
echo "torch working?"
python -c "import torch; print(torch.rand(5))"
echo
echo "mmagic working?"
python -c "import mmagic; print(mmagic.__version__)"
python -c "from mmagic.apis import MMagicInferencer"

[mcsmart76]

I was able to get things working with the addition of:

# Make sure we aren't caching wheels so mmcv builds the .so.
pip3 cache purge
pip3 install wheel

and changing mim install 'mmcv>=2.0.0,<=2.1.0' to:

# Use pip3 instead of mim to cause .so to be built.
#
# mmcv was bumped to 2.2.0 on 2024-04-16, which is incompatible
# with mmengine's version check as of 2024-04-25.
# https://github.com/open-mmlab/mmcv/pull/3087/commits
pip3 install 'mmcv>=2.0.0,<=2.1.0'

I also ran into #2110 for which I added:

pip3 install 'diffusers==0.24.0'

The full working script with a Python virtual environment is:

$ cat mmagic.sh
#!/bin/bash

VENV=~/.mmagic

# To cleanup, run `deactivate` and `rm -r $VENV`.

# Create my virtual environment at $VENV and activate it.
python3 -m venv $VENV
source $VENV/bin/activate

# Make sure we aren't caching wheels so mmcv builds the .so.
pip3 cache purge
pip3 install wheel

pip3 install torch
pip3 install torchvision
pip3 install -U openmim

# Use pip3 instead of mim to cause .so to be built.
#
# mmcv was bumped to 2.2.0 on 2024-04-16, which is incompatible
# with mmengine's version check as of 2024-04-25.
# https://github.com/open-mmlab/mmcv/pull/3087/commits
pip3 install 'mmcv>=2.0.0,<=2.1.0'

mim install mmengine
mim install mmagic

# https://github.com/open-mmlab/mmagic/issues/2110#issuecomment-1871812155
#
# Fixed "'No module named 'diffusers.pipelines.controlnet_xs'" issue by
# downgrading to 0.24.0.
pip3 install 'diffusers==0.24.0'

# Needed for the example at
# https://github.com/open-mmlab/mmagic/blob/main/docs/en/get_started/quick_run.md
pip3 install albumentations
pip3 install accelerate              

# Validate imports.
echo "torch working?"
python -c "import torch; print(torch.rand(5))"
echo
echo "mmagic working?"
python -c "import mmagic; print(mmagic.__version__)"
python -c "from mmagic.apis import MMagicInferencer"

echo
echo "source $VENV/bin/activate"