After the first epoch training, I am using del and gc.collect() on DataLoader, model, optimizer, everything except paths, and config, but the RAM is 6GB occupied after the first epoch. Then I am loading the model and creating dev and test DataLoaders to evaluate the model (7GB after), and then I delete test and dev DataLoaders, and RAM falls from 7GB to 6 GB. Then again it is growing during second epoch training and eventually, RAM is full.
I can’t upload the code (nevertheless it’s a quite huge project).
I’m sure no append() is used anywhere in my code (even if, it should be freed after gc.collect()?). I’m using .item() to get loss value.
Where can I look for the problems?