[Question] What typical saturation thresholds (as a rule of thumb) for each activation function

[github-abdellah-elyounsi]

Hi and thanks for this amazing work.

I'm working on the gradients and the importance of the initialization using this notebook. What are the typical saturation thresholds that would indicate my initialization might be problematic? I see that for tanh you have something like less than 10%. Would be a threshold for relu for example ?
How should the activations change over time ? keep the same mean and std ?
Thanks !