Code

distributed training#12

Code

Open

kolia wants to merge 47 commits intomainbeacon-biosignals/LighthouseFlux.jl:mainfrom ks/big_trainbeacon-biosignals/LighthouseFlux.jl:ks/big_trainCopy head branch name to clipboard

+498-136

Commits

Commits on Jul 19, 2020

• Remove Zygote.Params arg from loss_and_gradient

  Show description for d609538
  

  kolia

  committed
  d609538

  Copy full SHA for d609538

Commits on Jul 20, 2020

• Distributed train!

  

  kolia

  committed
  f92fdee

  Copy full SHA for f92fdee

• Bugfixes

  

  kolia

  committed
  88a14f9

  Copy full SHA for 88a14f9

• Remove duplicate method defs.

  

  kolia

  committed
  ded6689

  Copy full SHA for ded6689

Commits on Jul 21, 2020

• Differentiate calls to loss_and_gradient better.

  Show description for b403f9c
  

  kolia

  committed
  b403f9c

  Copy full SHA for b403f9c

• Bugfix attempt 2

  

  kolia

  committed
  3839b7a

  Copy full SHA for 3839b7a

• Bugfix attempt 3

  

  kolia

  committed
  d19379e

  Copy full SHA for d19379e

• Bugfix 4

  

  kolia

  committed
  c177e81

  Copy full SHA for c177e81

Commits on Jul 22, 2020

• Remove duplicate method defs.

  

  kolia

  committed
  16877fc

  Copy full SHA for 16877fc

• Adding straggler method back.

  

  kolia

  committed
  28d02e0

  Copy full SHA for 28d02e0

• buffered_batch_loader

  

  kolia

  committed
  954ace2

  Copy full SHA for 954ace2

Commits on Jul 23, 2020

• Align grads and params on their params orders.

  

  kolia

  committed
  2ee1666

  Copy full SHA for 2ee1666

• Bugfix

  

  kolia

  committed
  4942e32

  Copy full SHA for 4942e32

• Bugfix

  

  kolia

  committed
  702852b

  Copy full SHA for 702852b

Commits on Jul 24, 2020

• Bugfixes; occasional "CUDNNError: CUDNN_STATUS_NOT_INITIALIZED"

  

  kolia

  committed
  426a1c9

  Copy full SHA for 426a1c9

• Updated Flux/Zygote/CuArrays->CUDA

  Show description for 81a590b
  

  kolia

  committed
  81a590b

  Copy full SHA for 81a590b

• export buffered_batch_loader

  

  kolia

  committed
  b78825b

  Copy full SHA for b78825b

• add CUDA

  

  kolia

  committed
  11d7114

  Copy full SHA for 11d7114

Commits on Jul 25, 2020

• forgot to use CUDA

  

  kolia

  committed
  85db31d

  Copy full SHA for 85db31d

• Add Renormalizer optimiser

  

  kolia

  committed
  3cd605b

  Copy full SHA for 3cd605b

Commits on Jul 26, 2020

• Send less over wire to avoid trouble.

  

  kolia

  committed
  b618117

  Copy full SHA for b618117

• Log __plot_data__

  

  kolia

  committed
  1ee66a2

  Copy full SHA for 1ee66a2

Commits on Jul 28, 2020

• Bugfixes to robust controller-based training loop.

  

  kolia

  committed
  baed8e1

  Copy full SHA for baed8e1

• Don't crash if cannot kill stalled worker.

  

  kolia

  committed
  2b7bb5a

  Copy full SHA for 2b7bb5a

• Bump up timeout for worker returning a gradient.

  

  kolia

  committed
  ae527d9

  Copy full SHA for ae527d9

• Try not to get stuck in rmprocs()

  

  kolia

  committed
  59a4121

  Copy full SHA for 59a4121

• Merge remote-tracking branch 'origin' into ks/big_train

  

  kolia

  committed
  3ef496c

  Copy full SHA for 3ef496c

• Merge remote-tracking branch 'origin/ks/big_train' into ks/big_train

  

  kolia

  committed
  c602ec1

  Copy full SHA for c602ec1

• Handle case where no worker returns a gradient.

  

  kolia

  committed
  5e68d56

  Copy full SHA for 5e68d56

• Typo bugfix

  

  kolia

  committed
  0c6c5d5

  Copy full SHA for 0c6c5d5

• Whoups, nuked wrong one.

  

  kolia

  committed
  a3bf554

  Copy full SHA for a3bf554

Commits on Jul 29, 2020

• Remove probable blocker.

  Show description for ee2fe7a
  

  kolia

  committed
  ee2fe7a

  Copy full SHA for ee2fe7a

• Remove unresponsive workers

  

  kolia

  committed
  069daa4

  Copy full SHA for 069daa4

• Widen type of logger in loss_and_gradient

  

  kolia

  committed
  5fb0e53

  Copy full SHA for 5fb0e53

• Merge remote-tracking branch 'origin/ks/big_train' into ks/big_train

  

  kolia

  committed
  cf794a7

  Copy full SHA for cf794a7

• @show train_loss

  

  kolia

  committed
  5cf2d38

  Copy full SHA for 5cf2d38

Commits on Jul 31, 2020

• Tighten up memory consumption on master.

  

  kolia

  committed
  797767b

  Copy full SHA for 797767b

• Merge branch 'ks/big_train' of github.com:beacon-biosignals/LighthouseFlux.jl into ks/big_train

  

  kolia

  committed
  ccfabd2

  Copy full SHA for ccfabd2

• gpu_free_memory() util

  

  kolia

  committed
  3b10320

  Copy full SHA for 3b10320

• typo

  

  kolia

  committed
  810ea29

  Copy full SHA for 810ea29

Commits on Aug 4, 2020

• Preserve order / alignment.

  

  kolia

  committed
  44bdfd1

  Copy full SHA for 44bdfd1

Commits on Aug 8, 2020

• workers keep model between passes

  Show description for 82c2ee1
  

  kolia

  committed
  82c2ee1

  Copy full SHA for 82c2ee1

Commits on Aug 9, 2020

• can't assign to _model?

  

  kolia

  committed
  7143b82

  Copy full SHA for 7143b82

Commits on Aug 18, 2020

• Checkpoint before picking subset for PR.

  

  kolia

  committed
  cc482b9

  Copy full SHA for cc482b9

• Bugfix: apply! needs to be in module Flux.Optimise

  

  kolia

  committed
  e35b553

  Copy full SHA for e35b553

Commits on Sep 9, 2020

• Cleanup logger; it now lives in Lighthouse#general_logger

  

  kolia

  committed
  0ca38f6

  Copy full SHA for 0ca38f6

Commits on Oct 5, 2020

• avoid memory leak by calling GC on remote workers

  

  kolia

  committed
  6346ca6

  Copy full SHA for 6346ca6