Unittests to check model loading?

[Muennighoff]

There are lots of models that fail at the basic step of loading (e.g. #1689). Maybe it would be good to have a unit test that checks that all models can be loaded? One problem may the size of some large models, but maybe there is a way to work around that / we only test models that can be loaded in a reasonable amount of time

[isaac-chung]

Maybe we can limit these by n_parameters? Idk something like 2B.

[isaac-chung]

Adding to @Samoed 's suggestion, is it okay for us to run this test once / add as a script, instead of running this test for every PR? Tests are coming up to 47min.

[KennethEnevoldsen]

great idea,

generally think we could do a test that ensure that model can load (+ optionally additional checks like n_parameters match etc.). We coud then write these to a cache folder in repo (tests folder), which is invalidated based on a hash of the the model implementation (or model file). A simpler solution could also be to rerun after n-days? WDYT?

[isaac-chung]

Updated the PR checklist in #1698 to account for all suggestions 🙏