Add CNNs like ResNets, ConvNeXt, etc.

[yhli-ml]

Thank the authors and contributors for your novel, inspiring works. These works are well-presented.

Here is my question: I just wondered is there any possibility that SAE can be used to interpret the intermediate layer activations of sheer vision models like ResNet?

Compared to ViT, ResNet activations are far less discriminative, or to put it another way, far simpler. But as the authors found, “SAEs enable systematic observation of learned features, revealing fundamental differences between models”. ResNet should have its own traits, and what would it be like if we use SAE to understand it?

Additionally, I noticed that SAE should be trained on very large scale datasets, but why? If we train SAE only on very small scale datasets, say CIFAR-100, what would it be like?

(I understand that text-modal supervison tends to be important, so this is just a question out of nowhere and general or coarse-grained answers would be fine. Hope it does not look silly :D)

[samuelstevens]

Great question, and not silly at all! Yes, SAEs can be applied to CNNs. The canonical work for this topic is The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision by Liv Gorton. However, I have not seen follow-up work on this topic. If interpreting ResNets, ConvNeXt models, etc is important to your research, let me know, and I will turn this issue into a feature request for making saev architecture agnostic.

Training SAEs on small datasets would likely work. Plenty of prior work trains on less than 100M representation vectors (saev calls them "tokens" because it's exclusively ViTs right now). I'm not sure what would happen. Some thoughts:

• SAEs would learn a very limited set of features, with lots of dead latents, because there's not enough diversity to require all features
• If you solve the dead latents, SAEs would learn highly "split" features because there's not enough diversity, so instead of learning features like "red", "blue", "green", and "wing", "head", it would learn "red wing", "blue wing", "green wing", ... instead of more composable features. Perhaps Matryoshka SAEs would fix this.

[yhli-ml]

Thank you very much for your answer! Yes, interpreting ResNets, ConvNeXt models, etc is important to my research, hopefully we could turn this issue into a feature request, that would be wonderful.

Regarding what you mentioned, "SAEs would learn a very limited set of features, with lots of dead latents" when training with small datasets. -> Can we train a more light-weighted SAE, say decline the number of parameters from 65536 to 512, etc?

Thanks to your projects, I will try to implement this and try "Matryoshka SAEs" as you mentioned. Thank you again for your helpful comments, I'll close this issue.

[samuelstevens]

Sounds good. I'll re-open this as a feature request. saev currently does use matryoshka SAEs by default. You can see the implementation in

saev/src/saev/nn/objectives.py

Line 81 in 0f6b5b9

class MatryoshkaObjective(Objective):

.