How to solve timeout failure in eval?

[JiexingQi]

Hi, @tscholak . Sorry to bother you again. Recently I'm using PICARD with a large beam size (such as 8 or 12), and I always get this error when evaluating. I try to set

"val_max_time": 48000,

and it seems doesn't works. Could you kindly give me some suggestions to solve this problem? Thanks a lot. The error message are as follows:

***** Running Evaluation *****
  Num examples = 1300
  Batch size = 1
 31%|█████████████████████████████████████████████████▌                                                                                                             | 405/1300 [30:15<57:39,  3.87s/it]05/30/2022 08:54:16 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 213, 1246, 32101, 604, 3, 61, 1]
05/30/2022 08:54:16 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 41]
05/30/2022 08:54:16 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 213, 1246, 32100, 604, 3, 61, 1]
05/30/2022 08:54:26 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 41]
05/30/2022 08:54:26 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 213, 1246, 32101, 604, 3, 61, 3, 61]
05/30/2022 08:54:26 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 213, 1246, 32101, 604, 3, 61, 3, 201]
05/30/2022 08:54:36 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 201]
05/30/2022 08:54:47 - WARNING - seq2seq.utils.custom_picard_model_wrapper -   timeout failure: [0, 3490, 834, 9288, 15, 834, 15, 7480, 257, 1820, 1738, 690, 45, 3490, 213, 1246, 2490, 604, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 1768, 41, 1738, 690, 45, 3490, 563, 57, 690, 578, 3476, 41, 1429, 3, 61, 2490, 3274]