feature requests for RNNs

[CarloLucibello]

After the redesign in #2500, here is a list of potential improvements for recurrent layers and recurrent cells

• add an option in constructors to have trainable initial state
• use cuDNN
• implement the num_layers argument for stacked RNNs (issue Stacked RNN in Flux.jl?  #2452)
• add dropout (issue recurrent dropout  #1040)
• add bidirectional option (old PR Added the wrapper Bidirectional for RNN layers #1790)
• differentiate init for kernel and recurrent (Distinct init for kernel and recurrent #2522)

[MartinuzziFrancesco]

Could it be possible to add to the list the option to use different initializers for the input matrix and recurrent matrix? This is provided by both Keras/TF and Flax.

This should be as straightforward as

function RNNCell((in, out)::Pair, σ=relu;
    kernel_init = glorot_uniform,
    recurrent_kernel_init = glorot_uniform,
    bias = true)
    Wi = kernel_init(out, in)
    U = recurrent_kernel_init(out, out)
    b = create_bias(Wi, bias, size(Wi, 1))
    return RNNCell(σ, Wi, U, b)
end

I can also open a quick PR on this if needed

[CarloLucibello]

yes! PR welcome

[MartinuzziFrancesco]

Following up on this, should we also have an option to choose the init for the bias?

[CarloLucibello]

We don't do it for feedforward layers, if someone wants a non-zero bias can just change it manually in the constructor, layer.bias .= ...