Why can the weights of a zero-convolution layer obtain different gradients during training?

[sieve-github-access]

A common understanding in deep learning is that Initializing all the weights with zeros leads the neurons to learn the same features during training [1]. This is because all the weights get exactly the same gradient value and therefore receive exactly the same update during training. This is problematic because the layer effectively have only 1 neuron (aka channel) regardless how many neuron it has.

It seems to me that 1*1 zero-convolution layers, which are used throughout the paper, will suffer from this problem as well,.

That said, the huge success of ControlNet seems to imply that it doesn't suffer from this problem. Could anyone help me understand why?

[geroldmeisinger]

there is a link on the front page with an explanation, see here https://github.com/lllyasviel/ControlNet/blob/main/docs/faq.md

2h video from a guy explaining the paper https://www.youtube.com/watch?v=Mp-HMQcB_M4

[sieve-github-access]

The link doesn't explain why the weights of a zero-convolution layer won't be all the same , which is what this question is about ---
All weights starts with zero, receives the same gradient, so will they all be the same after training ?

[xiechun-tsukuba]

I am here to ask the same question.
the FAQ page only explains why the gradients are not zeros, but the real question is why the gradients are not the same for all the neurons in the zero initialized layer

[yeswecan]

@sieve-github-access @xiechun-tsukuba did you guys reach any conclusion on the topic? Sorry to recover the thread but I came here wondering the same thing

[xiechun-tsukuba]

@sieve-github-access @xiechun-tsukuba did you guys reach any conclusion on the topic? Sorry to recover the thread but I came here wondering the same thing

Hi, I asked several 'knowledgeable' people this question, and they all told me to read the FAQ page. However, when I explained why it did not answer my question, they went silent. Interesting.

Back to our question, I finally figured it out by writing out the gradient of one edge in the first (zero-initialized) layer. Assuming a simple MLP with one hidden layer and one output node, the gradient of the edge connecting input i and hidden node j is

where

You can see the gradient is depending on v_j and x_i. As v_j is not zero-initialized(imaging it is a replacement of the pretrained u-net), w_ij in the first layer will not have an identical gradient.

When will there be a problem? If we use the same initial values for each layer when initializing the network，v_j will be identical for every j, and the edge connecting the same input i will have an identical gradient.

I hope this helps others who have the same question.