Convolution with intermediate reduction

[ekhahniii]

ls it sensible to use cutlass to implement variants of convolution requiring intermediate reductions within the convolution window? An example could be modifying the usual dot product to be computed between a locally zero-mean window and kernel.

[hwu36]

Could you elaborate your example?

[ekhahniii]

Speaking in terms of convolutional neural networks, suppose a convolutional layer with some kernel $k$ is applied to an input. Then, the value at each location in the output can be written as the dot product between the kernel with the corresponding input window: $k \cdot x$.

Now, suppose instead we wish to compute the dot product not between $k$ and $x$, but between some modification to them based on the values within the kernel/input window. One example would be subtracting the mean of the kernel and the input window. Then, the desired output is $(k - \bar{k}) \cdot (x - \bar{x})$.

Is it sensible to use cutlass to achieve this?

[hwu36]

i still don't fully understand the example, especially how it works with NHWC layout.

However, I think computing mean on the fly is hard because it can involve cross threadblock communications.

[ekhahniii]

I'm interested primarily in variations on the typical definition of "convolution" in deep learning. The example above was intentionally vague w.r.t. implementation details, as I hoped to gain general insight and wanted to avoid biasing (at least initially) our discussion toward a particular implementation. 😄

However, I think computing mean on the fly is hard because it can involve cross threadblock communications.

This does seem to be an issue. Given that, would you agree it generally makes more sense to implement $(k-\bar{k}) \cdot (x-\bar{x})$ in multiple kernels?

[hwu36]

Yes, multiple kernels would be much easier.