FileNotFoundError: [Errno 2] No such file or directory

[marcomag416]

Hello, thank you for the useful tool.

I am getting the following error when trying to download laion400m:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nfs/home/magnanini/.pyenv/versions/3.10.17/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/nfs/home/magnanini/.pyenv/versions/3.10.17/lib/python3.10/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/nfs/home/magnanini/.pyenv/versions/3.10.17/lib/python3.10/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

To launch it, I use the command:

img2dataset --url_list data/metadata/laion_sampled --input_format "parquet"\
         --url_col "URL" --caption_col "TEXT" --output_format webdataset\
           --output_folder data/images/laion2m --processes_count 8 --thread_count 64 --image_size 256\
             --save_additional_columns '["NSFW","similarity"]' --enable_wandb True

I also tried using lower values for --processes_count and --thread_count, but I always get the same error.

Any suggestions?