TensorRT not found

[AlterCreator]

I use Google Colab.

I have problem with resampy,torchcrepe,and TensorRT.

I can fix resampy and torchcrepe wit just pip install.

But TensorRT is different,i has install it using pip install and it successfully.

But i still got same issues with TensorRT at train.py

Log during training:

2023-04-04 09:27:45.515902: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:27:45.515899: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

[Yeliaoyuan]

Got the same problem yesterday

There was no such problem last weekend

[liuhaottb]

Got the same problem

[cj0596]

Yes，and I got more error logs from yesterday,which was no such problem the day before yesterday.

2023-04-05 01:14:31.953845: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-05 01:14:32.886514: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
File "/content/so-vits-svc/train.py", line 24, in
import utils
File "/content/so-vits-svc/utils.py", line 20, in
from modules.crepe import CrepePitchExtractor
File "/content/so-vits-svc/modules/crepe.py", line 8, in
import torchcrepe
ModuleNotFoundError: No module named 'torchcrepe'

[Helloworld2345567]

i got the same problem too ,but it could train as usually if you don't care about it.

[AlterCreator]

i got the same problem too ,but it could train as usually if you don't care about it.

Where you got good pre-trained, i has trained it to epoch 6,4k and quality is not to good

[Helloworld2345567]

i got the same problem too ,but it could train as usually if you don't care about it.

Where you got good pre-trained, i has trained it to epoch 64k and quality is not to good

almost 1000 epoch , 20000steps. it has a clear voice(still has some electric murmur). Specifically,it's up to your trainset's quality.

[justinjohn0306]

Yes，and I got more error logs from yesterday,which was no such problem the day before yesterday.

2023-04-05 01:14:31.953845: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-04-05 01:14:32.886514: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Traceback (most recent call last): File "/content/so-vits-svc/train.py", line 24, in import utils File "/content/so-vits-svc/utils.py", line 20, in from modules.crepe import CrepePitchExtractor File "/content/so-vits-svc/modules/crepe.py", line 8, in import torchcrepe ModuleNotFoundError: No module named 'torchcrepe'

!pip install torchcrepe

[justinjohn0306]

use my nb: https://colab.research.google.com/github/justinjohn0306/so-vits-svc/blob/4.0/Sovits4_training_inference.ipynb

[voidtying]

use my nb: https://colab.research.google.com/github/justinjohn0306/so-vits-svc/blob/4.0/Sovits4_training_inference.ipynb

Thanks！

[cj0596]

use my nb: https://colab.research.google.com/github/justinjohn0306/so-vits-svc/blob/4.0/Sovits4_training_inference.ipynb

Thank You！But it seems another problem log appeared.I tried your colab file and the github colab file,all of them show the log below when I started training:

WARNING:44k:git hash values are different. f21f448(saved) != afe8c33(current)
2023-04-05 13:37:44.432704: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2023-04-05 13:37:45.714692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
DEBUG:h5py._conv:Creating converter from 7 to 5
DEBUG:h5py._conv:Creating converter from 5 to 7
DEBUG:h5py._conv:Creating converter from 7 to 5
DEBUG:h5py._conv:Creating converter from 5 to 7
DEBUG:jaxlib.mlir._mlir_libs:Initializing MLIR with module: _site_initialize_0
DEBUG:jaxlib.mlir._mlir_libs:Registering dialects from initializer <module 'jaxlib.mlir._mlir_libs._site_initialize_0' from '/usr/local/lib/python3.9/dist-packages/jaxlib/mlir/_mlir_libs/_site_initialize_0.so'>
DEBUG:jax._src.path:etils.epath found. Using etils.epath for file I/O.
INFO:numexpr.utils:NumExpr defaulting to 2 threads.
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
Traceback (most recent call last):
File "/content/so-vits-svc/train.py", line 315, in
main()
File "/content/so-vits-svc/train.py", line 53, in main
mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/content/so-vits-svc/train.py", line 70, in run
all_in_mem = hps.train.all_in_mem # If you have enough memory, turn on this option to avoid disk IO and speed up training.
AttributeError: 'HParams' object has no attribute 'all_in_mem'

[ioritree]

AttributeError: 'HParams' object has no attribute 'all_in_mem'
same problem

[ioritree]

use my nb: https://colab.research.google.com/github/justinjohn0306/so-vits-svc/blob/4.0/Sovits4_training_inference.ipynb

Thank You！But it seems another problem log appeared.I tried your colab file and the github colab file,all of them show the log below when I started training:

WARNING:44k:git hash values are different. f21f448(saved) != afe8c33(current) 2023-04-05 13:37:44.432704: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. 2023-04-05 13:37:45.714692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT DEBUG:h5py._conv:Creating converter from 7 to 5 DEBUG:h5py._conv:Creating converter from 5 to 7 DEBUG:h5py._conv:Creating converter from 7 to 5 DEBUG:h5py._conv:Creating converter from 5 to 7 DEBUG:jaxlib.mlir._mlir_libs:Initializing MLIR with module: _site_initialize_0 DEBUG:jaxlib.mlir._mlir_libs:Registering dialects from initializer <module 'jaxlib.mlir._mlir_libs._site_initialize_0' from '/usr/local/lib/python3.9/dist-packages/jaxlib/mlir/_mlir_libs/_site_initialize_0.so'> DEBUG:jax._src.path:etils.epath found. Using etils.epath for file I/O. INFO:numexpr.utils:NumExpr defaulting to 2 threads. INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. Traceback (most recent call last): File "/content/so-vits-svc/train.py", line 315, in main() File "/content/so-vits-svc/train.py", line 53, in main mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,)) File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 240, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 198, in start_processes while not context.join(): File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/content/so-vits-svc/train.py", line 70, in run all_in_mem = hps.train.all_in_mem # If you have enough memory, turn on this option to avoid disk IO and speed up training. AttributeError: 'HParams' object has no attribute 'all_in_mem'

old configs need add this line to work
"max_speclen": 512,
"port": "8001",
"keep_ckpts": 3,
"all_in_mem": false<<<<this
or will appear 'HParams' object has no attribute 'all_in_mem' error message

[Nerfixxxx000]

use my nb: https://colab.research.google.com/github/justinjohn0306/so-vits-svc/blob/4.0/Sovits4_training_inference.ipynb

Thank You！But it seems another problem log appeared.I tried your colab file and the github colab file,all of them show the log below when I started training:
WARNING:44k:git hash values are different. f21f448(saved) != afe8c33(current) 2023-04-05 13:37:44.432704: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. 2023-04-05 13:37:45.714692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT DEBUG:h5py._conv:Creating converter from 7 to 5 DEBUG:h5py._conv:Creating converter from 5 to 7 DEBUG:h5py._conv:Creating converter from 7 to 5 DEBUG:h5py._conv:Creating converter from 5 to 7 DEBUG:jaxlib.mlir._mlir_libs:Initializing MLIR with module: _site_initialize_0 DEBUG:jaxlib.mlir._mlir_libs:Registering dialects from initializer <module 'jaxlib.mlir._mlir_libs._site_initialize_0' from '/usr/local/lib/python3.9/dist-packages/jaxlib/mlir/_mlir_libs/_site_initialize_0.so'> DEBUG:jax._src.path:etils.epath found. Using etils.epath for file I/O. INFO:numexpr.utils:NumExpr defaulting to 2 threads. INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. Traceback (most recent call last): File "/content/so-vits-svc/train.py", line 315, in main() File "/content/so-vits-svc/train.py", line 53, in main mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,)) File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 240, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 198, in start_processes while not context.join(): File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/content/so-vits-svc/train.py", line 70, in run all_in_mem = hps.train.all_in_mem # If you have enough memory, turn on this option to avoid disk IO and speed up training. AttributeError: 'HParams' object has no attribute 'all_in_mem'

old configs need add this line to work "max_speclen": 512, "port": "8001", "keep_ckpts": 3, "all_in_mem": false<<<<this or will appear 'HParams' object has no attribute 'all_in_mem' error message

Update your configs file with reference to config_template.json. or in an existing config file add
"all_in_mem": false (under "keep_ckpts": 3)

[Akarin0v0]

use my nb: https://colab.research.google.com/github/justinjohn0306/so-vits-svc/blob/4.0/Sovits4_training_inference.ipynb

Thank You！But it seems another problem log appeared.I tried your colab file and the github colab file,all of them show the log below when I started training:
WARNING:44k:git hash values are different. f21f448(saved) != afe8c33(current) 2023-04-05 13:37:44.432704: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. 2023-04-05 13:37:45.714692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT DEBUG:h5py._conv:Creating converter from 7 to 5 DEBUG:h5py._conv:Creating converter from 5 to 7 DEBUG:h5py._conv:Creating converter from 7 to 5 DEBUG:h5py._conv:Creating converter from 5 to 7 DEBUG:jaxlib.mlir._mlir_libs:Initializing MLIR with module: _site_initialize_0 DEBUG:jaxlib.mlir._mlir_libs:Registering dialects from initializer <module 'jaxlib.mlir._mlir_libs._site_initialize_0' from '/usr/local/lib/python3.9/dist-packages/jaxlib/mlir/_mlir_libs/_site_initialize_0.so'> DEBUG:jax._src.path:etils.epath found. Using etils.epath for file I/O. INFO:numexpr.utils:NumExpr defaulting to 2 threads. INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. Traceback (most recent call last): File "/content/so-vits-svc/train.py", line 315, in main() File "/content/so-vits-svc/train.py", line 53, in main mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,)) File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 240, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 198, in start_processes while not context.join(): File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/content/so-vits-svc/train.py", line 70, in run all_in_mem = hps.train.all_in_mem # If you have enough memory, turn on this option to avoid disk IO and speed up training. AttributeError: 'HParams' object has no attribute 'all_in_mem'

old configs need add this line to work "max_speclen": 512, "port": "8001", "keep_ckpts": 3, "all_in_mem": false<<<<this or will appear 'HParams' object has no attribute 'all_in_mem' error message

Update your configs file with reference to config_template.json. or in an existing config file add "all_in_mem": false (under "keep_ckpts": 3)

problem solved，thanks a lot :)!

[cj0596]

Thanks!