Model is hard to be modified

Hey,

just wondering if anyone else has run into the same issue when trying to modify the model—for example, adding new trainable modules or plugging in extra encoders. I’ve found it really hard to get the model to converge after these changes. It seems to overfit almost exclusively to the high-compression autoencoder.