Expanding Weight Initialization Module #3
LazaroHurtado
started this conversation in
Ideas
Replies: 2 comments
-
I'll try my hand at implementing He Initialization tomorrow. |
Beta Was this translation helpful? Give feedback.
0 replies
-
I will be implementing LeCun weight initialization. Please let me know if this causes any conflicts. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Using the proper weight initialization can significantly boost performance gains by accelerating convergence of SGD. With this being said, we should expand our
WeightInit
enum by incorporating the He and LeCun schemes. Currently, we only offer Glorot/Xavier but this is suboptimal for the ReLu activation function so some tiny changes must be made.Anyone can claim implementing these features:
Here are some extra resources you might find helpful:
Glorot and He discussion
wandb Intro to Weight Inits
Beta Was this translation helpful? Give feedback.
All reactions