Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to build transformer-engine #1270

Open
jaefan11 opened this issue Oct 18, 2024 · 3 comments
Open

Failed to build transformer-engine #1270

jaefan11 opened this issue Oct 18, 2024 · 3 comments
Labels
build Build system

Comments

@jaefan11
Copy link

pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable

Thanks in advance.

python: 3.9
pytorch: 2.1.0
cuda: 12.1
gcc: 7.5.0

FAILED: CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o
/usr/local/bin/c++ -DNV_CUDNN_FRONTEND_USE_DYNAMIC_LOADING -Dtransformer_engine_EXPORTS -I/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/.. -I/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/include -I/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/../../3rdparty/cudnn-frontend/include -I/tmp/pip-req-build-hfgbdcj5/build/cmake/string_headers -isystem /usr/local/cuda-12.1/targets/x86_64-linux/include -Wl,--version-script=/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/libtransformer_engine.version -O3 -DNDEBUG -std=gnu++1z -fPIC -MD -MT CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o -MF CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o.d -o CMakeFiles/transformer_engine.dir/util/cuda_driver.cpp.o -c /tmp/pip-req-build-hfgbdcj5/transformer_engine/common/util/cuda_driver.cpp
/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/util/cuda_driver.cpp:9:10: fatal error: filesystem: No such file or directory
#include
^~~~~~~~~~~~
compilation terminated.
[3/36] /usr/local/bin/c++ -DNV_CUDNN_FRONTEND_USE_DYNAMIC_LOADING -Dtransformer_engine_EXPORTS -I/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/.. -I/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/include -I/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/../../3rdparty/cudnn-frontend/include -I/tmp/pip-req-build-hfgbdcj5/build/cmake/string_headers -isystem /usr/local/cuda-12.1/targets/x86_64-linux/include -Wl,--version-script=/tmp/pip-req-build-hfgbdcj5/transformer_engine/common/libtransformer_engine.version -O3 -DNDEBUG -std=gnu++1z -fPIC -MD -MT CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o -MF CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o.d -o CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o -c /tmp/pip-req-build-hfgbdcj5/transformer_engine/common/util/cuda_runtime.cpp
FAILED: CMakeFiles/transformer_engine.dir/util/cuda_runtime.cpp.o
@timmoon10
Copy link
Collaborator

timmoon10 commented Oct 18, 2024 •
edited
Loading

Try updating your compiler. TE requires C++17 and GCC added the filesystem header in GCC 8.1 (see GCC's C++17 support). We currently use GCC 13.2.0 in our internal builds.

@timmoon10 timmoon10 added the build Build system label Oct 18, 2024
@s-smits
Copy link

s-smits commented Nov 5, 2024

python: 3.11
pytorch: 2.4.1
cuda: 12.4
cudnn: 9
gcc: 11.4.0

docker: pytorch/pytorch:2.4.1-cuda12.4-cudnn9-devel
RTX4090

Collecting git+https://github.com/NVIDIA/TransformerEngine.git@stable
  Cloning https://github.com/NVIDIA/TransformerEngine.git (to revision stable) to /tmp/pip-req-build-s96o7cy6
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA/TransformerEngine.git /tmp/pip-req-build-s96o7cy6
  Running command git checkout -b stable --track origin/stable
  Switched to a new branch 'stable'
  Branch 'stable' set up to track remote branch 'stable' from 'origin'.
  Resolved https://github.com/NVIDIA/TransformerEngine.git to commit c27ee60ec746210bcea4ec33958dbbff06706506
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... done
Collecting pydantic (from transformer_engine==1.11.0+c27ee60)
  Using cached pydantic-2.9.2-py3-none-any.whl.metadata (149 kB)
Collecting flash-attn!=2.0.9,!=2.1.0,<=2.6.3,>=2.0.6 (from transformer_engine==1.11.0+c27ee60)
  Downloading flash_attn-2.6.3.tar.gz (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 4.9 MB/s eta 0:00:00a 0:00:01
  Preparing metadata (setup.py) ... done
Requirement already satisfied: torch in /opt/conda/lib/python3.11/site-packages (from transformer_engine==1.11.0+c27ee60) (2.4.1+cu124)
Collecting importlib-metadata>=1.0 (from transformer_engine==1.11.0+c27ee60)
  Using cached importlib_metadata-8.5.0-py3-none-any.whl.metadata (4.8 kB)
Requirement already satisfied: packaging in /opt/conda/lib/python3.11/site-packages (from transformer_engine==1.11.0+c27ee60) (24.1)
Requirement already satisfied: einops in /opt/conda/lib/python3.11/site-packages (from flash-attn!=2.0.9,!=2.1.0,<=2.6.3,>=2.0.6->transformer_engine==1.11.0+c27ee60) (0.8.0)
Requirement already satisfied: zipp>=3.20 in /opt/conda/lib/python3.11/site-packages (from importlib-metadata>=1.0->transformer_engine==1.11.0+c27ee60) (3.20.1)
Collecting annotated-types>=0.6.0 (from pydantic->transformer_engine==1.11.0+c27ee60)
  Using cached annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.23.4 (from pydantic->transformer_engine==1.11.0+c27ee60)
  Using cached pydantic_core-2.23.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: typing-extensions>=4.6.1 in /opt/conda/lib/python3.11/site-packages (from pydantic->transformer_engine==1.11.0+c27ee60) (4.12.2)
Requirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (3.15.4)
Requirement already satisfied: sympy in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (1.13.2)
Requirement already satisfied: networkx in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (3.3)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (3.1.4)
Requirement already satisfied: fsspec in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (2024.9.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.99 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.4.99)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.99 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.4.99)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.99 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.4.99)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.2.65 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.4.2.65)
Requirement already satisfied: nvidia-cufft-cu12==11.2.0.44 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (11.2.0.44)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.119 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (10.3.5.119)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.0.99 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (11.6.0.99)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.0.142 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.3.0.142)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.99 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.4.99)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.99 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (12.4.99)
Requirement already satisfied: triton==3.0.0 in /opt/conda/lib/python3.11/site-packages (from torch->transformer_engine==1.11.0+c27ee60) (3.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->torch->transformer_engine==1.11.0+c27ee60) (2.1.5)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from sympy->torch->transformer_engine==1.11.0+c27ee60) (1.3.0)
Using cached importlib_metadata-8.5.0-py3-none-any.whl (26 kB)
Using cached pydantic-2.9.2-py3-none-any.whl (434 kB)
Using cached pydantic_core-2.23.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB)
Building wheels for collected packages: transformer_engine, flash-attn
  Building wheel for transformer_engine (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [425 lines of output]
      WARNING: Skipping transformer_engine_cu12 as it is not installed.
      WARNING: Skipping transformer_engine_torch as it is not installed.
      WARNING: Skipping transformer_engine_paddle as it is not installed.
      WARNING: Skipping transformer_engine_jax as it is not installed.
      WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
      /opt/conda/lib/python3.11/site-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!
      
              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
              ********************************************************************************
      
      !!
        dist.fetch_build_eggs(dist.setup_requires)
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-311/transformer_engine
      copying transformer_engine/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/common
      copying transformer_engine/common/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/common
      copying transformer_engine/common/utils.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/common
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/dot.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/fp8.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/layernorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/setup.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/sharding.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      copying transformer_engine/jax/attention.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/constants.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/distributed.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/fp8.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/profile.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/recompute.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/setup.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/utils.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/cpp_extensions.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      copying transformer_engine/paddle/fp8_buffer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/constants.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/export.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/float8_tensor.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/fp8.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/numerics_debug.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/attention.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/cpu_offload.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/distributed.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/graph.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/jit.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/permutation.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/setup.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/te_onnx_extensions.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      copying transformer_engine/pytorch/utils.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/common/recipe
      copying transformer_engine/common/recipe/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/common/recipe
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/base.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/activation.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/attention.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/custom_call.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/misc.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/normalization.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/quantization.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      copying transformer_engine/jax/cpp_extensions/transpose.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/cpp_extensions
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
      copying transformer_engine/jax/flax/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
      copying transformer_engine/jax/flax/module.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
      copying transformer_engine/jax/flax/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
      copying transformer_engine/jax/praxis/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
      copying transformer_engine/jax/praxis/module.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
      copying transformer_engine/jax/praxis/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/base.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/layernorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/layernorm_linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/rmsnorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/attention.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      copying transformer_engine/paddle/layer/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/activation.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/cast.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/normalization.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/padding.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/transpose.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/_common.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/fused_attn.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      copying transformer_engine/pytorch/cpp_extensions/gemm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/_common.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/base.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/fp8_padding.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/fp8_unpadding.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/grouped_linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/layernorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/layernorm_linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      copying transformer_engine/pytorch/module/rmsnorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      copying transformer_engine/pytorch/ops/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      copying transformer_engine/pytorch/ops/_common.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      copying transformer_engine/pytorch/ops/fuser.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      copying transformer_engine/pytorch/ops/linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      copying transformer_engine/pytorch/ops/op.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      copying transformer_engine/pytorch/ops/sequential.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/optimizers
      copying transformer_engine/pytorch/optimizers/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/optimizers
      copying transformer_engine/pytorch/optimizers/multi_tensor_apply.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/optimizers
      copying transformer_engine/pytorch/optimizers/fused_adam.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/optimizers
      copying transformer_engine/pytorch/optimizers/fused_sgd.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/optimizers
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/tensor
      copying transformer_engine/pytorch/tensor/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/tensor
      copying transformer_engine/pytorch/tensor/float8_tensor.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/tensor
      copying transformer_engine/pytorch/tensor/quantized_tensor.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/tensor
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/add_in_place.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/all_gather.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/all_reduce.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/identity.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/make_extra_output.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/reduce_scatter.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/reshape.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/basic_linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      copying transformer_engine/pytorch/ops/basic/bias.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/basic
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/fused
      copying transformer_engine/pytorch/ops/fused/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/fused
      copying transformer_engine/pytorch/ops/fused/backward_linear_add.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/fused
      copying transformer_engine/pytorch/ops/fused/forward_linear_bias_activation.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/fused
      copying transformer_engine/pytorch/ops/fused/forward_linear_bias_add.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/ops/fused
      running egg_info
      creating transformer_engine.egg-info
      writing transformer_engine.egg-info/PKG-INFO
      writing dependency_links to transformer_engine.egg-info/dependency_links.txt
      writing requirements to transformer_engine.egg-info/requires.txt
      writing top-level names to transformer_engine.egg-info/top_level.txt
      writing manifest file 'transformer_engine.egg-info/SOURCES.txt'
      reading manifest file 'transformer_engine.egg-info/SOURCES.txt'
      adding license file 'LICENSE'
      writing manifest file 'transformer_engine.egg-info/SOURCES.txt'
      /opt/conda/lib/python3.11/site-packages/setuptools/command/build_py.py:220: _Warning: Package 'transformer_engine.pytorch.csrc' is absent from the `packages` configuration.
      !!
      
              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'transformer_engine.pytorch.csrc' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.
      
              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'transformer_engine.pytorch.csrc' is explicitly added
              to the `packages` configuration field.
      
              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).
      
              You can read more about "package discovery" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
      
              If you don't want 'transformer_engine.pytorch.csrc' to be distributed and are
              already explicitly excluding 'transformer_engine.pytorch.csrc' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.
      
              You can read more about "package data files" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
      
      
              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************
      
      !!
        check.warn(importable)
      /opt/conda/lib/python3.11/site-packages/setuptools/command/build_py.py:220: _Warning: Package 'transformer_engine.pytorch.csrc.extensions' is absent from the `packages` configuration.
      !!
      
              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'transformer_engine.pytorch.csrc.extensions' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.
      
              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'transformer_engine.pytorch.csrc.extensions' is explicitly added
              to the `packages` configuration field.
      
              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).
      
              You can read more about "package discovery" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
      
              If you don't want 'transformer_engine.pytorch.csrc.extensions' to be distributed and are
              already explicitly excluding 'transformer_engine.pytorch.csrc.extensions' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.
      
              You can read more about "package data files" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
      
      
              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************
      
      !!
        check.warn(importable)
      /opt/conda/lib/python3.11/site-packages/setuptools/command/build_py.py:220: _Warning: Package 'transformer_engine.pytorch.csrc.extensions.multi_tensor' is absent from the `packages` configuration.
      !!
      
              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'transformer_engine.pytorch.csrc.extensions.multi_tensor' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.
      
              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'transformer_engine.pytorch.csrc.extensions.multi_tensor' is explicitly added
              to the `packages` configuration field.
      
              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).
      
              You can read more about "package discovery" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
      
              If you don't want 'transformer_engine.pytorch.csrc.extensions.multi_tensor' to be distributed and are
              already explicitly excluding 'transformer_engine.pytorch.csrc.extensions.multi_tensor' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.
      
              You can read more about "package data files" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
      
      
              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************
      
      !!
        check.warn(importable)
      /opt/conda/lib/python3.11/site-packages/setuptools/command/build_py.py:220: _Warning: Package 'transformer_engine.pytorch.csrc.userbuffers' is absent from the `packages` configuration.
      !!
      
              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'transformer_engine.pytorch.csrc.userbuffers' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.
      
              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'transformer_engine.pytorch.csrc.userbuffers' is explicitly added
              to the `packages` configuration field.
      
              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).
      
              You can read more about "package discovery" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
      
              If you don't want 'transformer_engine.pytorch.csrc.userbuffers' to be distributed and are
              already explicitly excluding 'transformer_engine.pytorch.csrc.userbuffers' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.
      
              You can read more about "package data files" on setuptools documentation page:
      
              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
      
      
              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************
      
      !!
        check.warn(importable)
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc
      copying transformer_engine/pytorch/csrc/common.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc
      copying transformer_engine/pytorch/csrc/ts_fp8_op.cpp -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/activation.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/apply_rope.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/attention.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/cast.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/gemm.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/misc.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/normalization.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/padding.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/permutation.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/pybind.cpp -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/recipe.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/softmax.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      copying transformer_engine/pytorch/csrc/extensions/transpose.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions/multi_tensor
      copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_adam.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions/multi_tensor
      copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_l2norm_kernel.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions/multi_tensor
      copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_scale_kernel.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions/multi_tensor
      copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_sgd_kernel.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/extensions/multi_tensor
      creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/userbuffers
      copying transformer_engine/pytorch/csrc/userbuffers/ipcsocket.cc -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/userbuffers
      copying transformer_engine/pytorch/csrc/userbuffers/userbuffers-host.cpp -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/userbuffers
      copying transformer_engine/pytorch/csrc/userbuffers/userbuffers.cu -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/csrc/userbuffers
      running build_ext
      Building CMake extension transformer_engine
      Running command /opt/conda/bin/cmake -S /tmp/pip-req-build-s96o7cy6/transformer_engine/common -B /tmp/pip-req-build-s96o7cy6/build/cmake -DPython_EXECUTABLE=/opt/conda/bin/python3.11 -DPython_INCLUDE_DIR=/opt/conda/include/python3.11 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-s96o7cy6/build/lib.linux-x86_64-cpython-311 -DCMAKE_CUDA_ARCHITECTURES=70;80;89;90 -Dpybind11_DIR=/tmp/pip-req-build-s96o7cy6/.eggs/pybind11-2.13.6-py3.11.egg/pybind11/share/cmake/pybind11 -GNinja
      -- The CUDA compiler identification is NVIDIA 12.4.131
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting CUDA compiler ABI info
      -- Detecting CUDA compiler ABI info - done
      -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
      -- Detecting CUDA compile features
      -- Detecting CUDA compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found CUDAToolkit: /usr/local/cuda/targets/x86_64-linux/include (found version "12.4.131")
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      CMake Error at /tmp/pip-req-build-s96o7cy6/3rdparty/cudnn-frontend/cmake/cuDNN.cmake:3 (find_path):
        Could not find CUDNN_INCLUDE_DIR using the following files: cudnn.h
      Call Stack (most recent call first):
        CMakeLists.txt:40 (include)
      
      
      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/pip-req-build-s96o7cy6/build_tools/build_ext.py", line 89, in _build_cmake
          subprocess.run(command, cwd=build_dir, check=True)
        File "/opt/conda/lib/python3.11/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['/opt/conda/bin/cmake', '-S', '/tmp/pip-req-build-s96o7cy6/transformer_engine/common', '-B', '/tmp/pip-req-build-s96o7cy6/build/cmake', '-DPython_EXECUTABLE=/opt/conda/bin/python3.11', '-DPython_INCLUDE_DIR=/opt/conda/include/python3.11', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-s96o7cy6/build/lib.linux-x86_64-cpython-311', '-DCMAKE_CUDA_ARCHITECTURES=70;80;89;90', '-Dpybind11_DIR=/tmp/pip-req-build-s96o7cy6/.eggs/pybind11-2.13.6-py3.11.egg/pybind11/share/cmake/pybind11', '-GNinja']' returned non-zero exit status 1.
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-s96o7cy6/setup.py", line 174, in <module>
          setuptools.setup(
        File "/opt/conda/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 183, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
          dist.run_commands()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
          self.run_command(cmd)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-req-build-s96o7cy6/setup.py", line 53, in run
          super().run()
        File "/opt/conda/lib/python3.11/site-packages/wheel/_bdist_wheel.py", line 378, in run
          self.run_command("build")
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
          super().run_command(command)
        File "/opt/conda/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-req-build-s96o7cy6/build_tools/build_ext.py", line 119, in run
          ext._build_cmake(
        File "/tmp/pip-req-build-s96o7cy6/build_tools/build_ext.py", line 91, in _build_cmake
          raise RuntimeError(f"Error when running CMake: {e}")
      RuntimeError: Error when running CMake: Command '['/opt/conda/bin/cmake', '-S', '/tmp/pip-req-build-s96o7cy6/transformer_engine/common', '-B', '/tmp/pip-req-build-s96o7cy6/build/cmake', '-DPython_EXECUTABLE=/opt/conda/bin/python3.11', '-DPython_INCLUDE_DIR=/opt/conda/include/python3.11', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-s96o7cy6/build/lib.linux-x86_64-cpython-311', '-DCMAKE_CUDA_ARCHITECTURES=70;80;89;90', '-Dpybind11_DIR=/tmp/pip-req-build-s96o7cy6/.eggs/pybind11-2.13.6-py3.11.egg/pybind11/share/cmake/pybind11', '-GNinja']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for transformer_engine
  Running setup.py clean for transformer_engine
  Building wheel for flash-attn (setup.py) ... done
  Created wheel for flash-attn: filename=flash_attn-2.6.3-cp311-cp311-linux_x86_64.whl size=187328293 sha256=25405479af3f6865c873ee3bbdfadcfccea9055355de78e7cd6b93170e9d4377
  Stored in directory: /root/.cache/pip/wheels/e3/ef/b1/7889928ffa2dea61032e61480db4e4c20d00a9d9e28cd4f55a
Successfully built flash-attn
Failed to build transformer_engine
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (transformer_engine)

@timmoon10
Copy link
Collaborator

@s-smits CMake can't find cuDNN. Please set CUDNN_PATH in the environment. See #355 (comment) for more guidance on common build errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
build Build system
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@timmoon10 @jaefan11 @s-smits