NumericalFeatureTokenizer after train test split

[OrtalDayan]

Looking at the code it seems to me that it doesn't matter (won't cause data leakage ) if we apply the NumericalFeatureTokenizer or NumEmbeddings class before or after train test split. Am I right?

[Yura52]

These are PyTorch modules in the same way as, for example, torch.nn.Linear is. It means that:

• they are end-to-end trainable parts of a model, not preprocessing tools
• they are always "applied" to the data, not just once (as with preprocessing tools such as sklearn.preprocessing.StandardScaler)

Does this help?

[OrtalDayan]

Yes. Thank you very much.