Why doesn't the loss decrease during training？

[taoumb]

Can this code be trained correctly?

[xy-lin77]

I have run 50,000 iterations, it seems that the loss is always between 1.2-1.4

[EdoardoBotta]

Are you using the default hyperparameters for the model size? If yes, those generate a very small model and I do not expect to do well. I am planning to experiment with bigger model size to ensure the model can be trained.

[xy-lin77]

Thank you for your reply. I will try the other hyperparameters.

[TobinShaw]

@EdoardoBotta @xy-lin77 @taoumb I also encountered this issue. Have you made any new attempts and got some findings?

[EdoardoBotta]

I have updated the model size and dataset with a3812ff, making the model size the same as the original one from the paper and adding a textual embedding of the movie title as additional features. The loss reflected in the progress bar is now a moving average and it steadily decreases during execution of the training loop.