Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3 run.py failed #5

Open
SeekPoint opened this issue May 7, 2023 · 2 comments
Open

python3 run.py failed #5

SeekPoint opened this issue May 7, 2023 · 2 comments

Comments

@SeekPoint
Copy link

(gh_baize-chatbot) ub2004@ub2004-B85M-A0:~/llm_dev/RWKV-CUDA/wkv$ python3 run.py
Using /home/ub2004/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Creating extension directory /home/ub2004/.cache/torch_extensions/py38_cu117/wkv...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/ub2004/.cache/torch_extensions/py38_cu117/wkv/build.ninja...
Building extension module wkv...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 /wd4624 -c /home/ub2004/llm_dev/RWKV-CUDA/wkv/cuda/wkv_op.cpp -o wkv_op.o
FAILED: wkv_op.o
c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 /wd4624 -c /home/ub2004/llm_dev/RWKV-CUDA/wkv/cuda/wkv_op.cpp -o wkv_op.o
c++: error: /wd4624: No such file or directory
[2/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/TH -isystem /home/ub2004/.local/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' --use_fast_math --extra-device-vectorization -std=c++17 -c /home/ub2004/llm_dev/RWKV-CUDA/wkv/cuda/wkv_cuda_v2.cu -o wkv_cuda_v2.cuda.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "run.py", line 86, in
wkv_cuda = load(name="wkv", sources=["cuda/wkv_op.cpp", f"cuda/wkv_cuda_v{CUDA_KERNEL_VERSION}.cu"],
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/ub2004/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'wkv'
(gh_baize-chatbot) ub2004@ub2004-B85M-A0:~/llm_dev/RWKV-CUDA/wkv$
@Vectorrent
Copy link

Same issue here, when trying to compile inside of the official NVIDIA Docker image:

 => ERROR [vtx 15/16] RUN python3 /vtx/lab/rwkv-cuda/wkv/run.py                                                                                  20.2s
------
 > [vtx 15/16] RUN python3 /vtx/lab/rwkv-cuda/wkv/run.py:
#0 1.852 No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
#0 1.853 Using /root/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
#0 1.853 Creating extension directory /root/.cache/torch_extensions/py310_cu117/wkv...
#0 1.905 Detected CUDA files, patching ldflags
#0 1.905 Emitting ninja build file /root/.cache/torch_extensions/py310_cu117/wkv/build.ninja...
#0 1.905 Building extension module wkv...
#0 1.905 Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
#0 2.947 [1/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --use_fast_math --extra-device-vectorization -std=c++14 -c /vtx/lab/rwkv-cuda/wkv/cuda/wkv_cuda_v2.cu -o wkv_cuda_v2.cuda.o 
#0 19.14 [2/3] c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 /wd4624 -c /vtx/lab/rwkv-cuda/wkv/cuda/wkv_op.cpp -o wkv_op.o 
#0 19.14 FAILED: wkv_op.o 
#0 19.14 c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 /wd4624 -c /vtx/lab/rwkv-cuda/wkv/cuda/wkv_op.cpp -o wkv_op.o 
#0 19.14 c++: warning: /wd4624: linker input file unused because linking not done
#0 19.14 c++: error: /wd4624: linker input file not found: No such file or directory
#0 19.14 ninja: build stopped: subcommand failed.
#0 19.17 Traceback (most recent call last):
#0 19.17   File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
#0 19.17     subprocess.run(
#0 19.17   File "/usr/lib/python3.10/subprocess.py", line 524, in run
#0 19.18     raise CalledProcessError(retcode, process.args,
#0 19.18 subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
#0 19.18 
#0 19.18 The above exception was the direct cause of the following exception:
#0 19.18 
#0 19.18 Traceback (most recent call last):
#0 19.18   File "/vtx/lab/rwkv-cuda/wkv/run.py", line 86, in <module>
#0 19.18     wkv_cuda = load(name="wkv", sources=["cuda/wkv_op.cpp", f"cuda/wkv_cuda_v{CUDA_KERNEL_VERSION}.cu"],
#0 19.18   File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1284, in load
#0 19.18     return _jit_compile(
#0 19.18   File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
#0 19.18     _write_ninja_file_and_build_library(
#0 19.18   File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
#0 19.18     _run_ninja_build(
#0 19.18   File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
#0 19.18     raise RuntimeError(message) from e
#0 19.18 RuntimeError: Error building extension 'wkv'
------
failed to solve: process "/bin/sh -c python3 /vtx/lab/rwkv-cuda/wkv/run.py" did not complete successfully: exit code: 1

@Chengyui
Copy link

I fixed the issue by removing "extra_cflags=['/wd4624'] " while my device is on Ubuntu 20.04.
See #2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SeekPoint @Chengyui @Vectorrent