Better logging with joblib.Parallel

[olliethomas]

current output

[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    4.0s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    7.2s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:   10.9s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed:   15.3s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:   18.6s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   6 out of   6 | elapsed:   22.2s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   7 out of   7 | elapsed:   22.2s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   8 out of   8 | elapsed:   22.2s remaining:    0.0s
[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:   22.3s remaining:    0.0s

It must be possible to populate "out of n' and an estimate of remaining time

[olliethomas]

these do get set when num_jobs>1

[Parallel(n_jobs=128)]: Using backend LokyBackend with 128 concurrent workers.
[Parallel(n_jobs=128)]: Done   7 out of  70 | elapsed:    5.1s remaining:   45.7s
[Parallel(n_jobs=128)]: Done  15 out of  70 | elapsed:    5.3s remaining:   19.3s

🕵️

[tmke8]

So, you're happy with the output for num_jobs>1?

[olliethomas]

So, you're happy with the output for num_jobs>1?

Sometimes. With large number of jobs the following occurs

[Parallel(n_jobs=18)]: Using backend LokyBackend with 18 concurrent workers.                                                                                                                                                                                                                    
[Parallel(n_jobs=18)]: Done   5 tasks      | elapsed:    3.9s                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  14 tasks      | elapsed:  1.5min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  25 tasks      | elapsed:  1.6min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  36 tasks      | elapsed:  3.2min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  49 tasks      | elapsed:  4.4min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  62 tasks      | elapsed:  5.0min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  77 tasks      | elapsed:  6.3min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done  92 tasks      | elapsed:  7.2min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done 109 tasks      | elapsed:  8.2min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done 126 tasks      | elapsed:  9.3min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done 145 tasks      | elapsed: 10.7min                                                                                                                                                                                                                                   
[Parallel(n_jobs=18)]: Done 164 tasks      | elapsed: 12.0min
[Parallel(n_jobs=18)]: Done 185 tasks      | elapsed: 13.1min
[Parallel(n_jobs=18)]: Done 206 tasks      | elapsed: 13.2min
[Parallel(n_jobs=18)]: Done 229 tasks      | elapsed: 13.3min
[Parallel(n_jobs=18)]: Done 252 tasks      | elapsed: 13.3min
[Parallel(n_jobs=18)]: Done 277 tasks      | elapsed: 13.4min
[Parallel(n_jobs=18)]: Done 302 tasks      | elapsed: 13.4min
[Parallel(n_jobs=18)]: Done 329 tasks      | elapsed: 13.4min
[Parallel(n_jobs=18)]: Done 356 tasks      | elapsed: 13.5min
[Parallel(n_jobs=18)]: Done 385 tasks      | elapsed: 13.5min
[Parallel(n_jobs=18)]: Done 414 tasks      | elapsed: 18.3min
[Parallel(n_jobs=18)]: Done 445 tasks      | elapsed: 27.2min
[Parallel(n_jobs=18)]: Done 476 tasks      | elapsed: 42.3min
[Parallel(n_jobs=18)]: Done 509 tasks      | elapsed: 49.1min
[Parallel(n_jobs=18)]: Done 542 tasks      | elapsed: 60.5min
[Parallel(n_jobs=18)]: Done 600 out of 600 | elapsed: 76.7min finished