FNet in TTS Encoders

[manhph2211]

Dear @rishikksh20 ,

Thanks for your implementation! Can I ask what do you think about the effectiveness of Furier features as well as FNet when it's used to replace multi-head attention in TTS encoders which is quite a concrete approach so far! I'm thinking about if it can help to reduce the complexity of the TTS models but still maintain good performance 🤔

Thanks!

Max