Memory leak

[HunorLaczko]

Hi,

My dataset is only around 25gb in memory, but after training for a few hours the memory usage is already more than 100gb and it keeps slowly but constantly increasing. I'm using a custom dataset created with tf.data.Dataset.from_generator. Do you know where the problem could be?

Using 1 gpu (3090), with custom cuda, batch of 4, no labels. The rest is more or less the original code.