It’s the CPU memory that is blowing, the GPU memory stays stable.
I am not caching anything inside the model, only torch.nn ops. I will try gc.collect() and let you know. Thanks for the advice, I am not sure I understood the pinned memory feature, can it be related to my problem somehow ?
To give you an idea, in test mode I am around 9 GB used, in training mode it’s around 20 GB. The thing is that memory usage in test mode is stable between 8-9 GB, but during training the RAM is slowly eatean, batcth after batch, constantly growing until the next test step where it goes back to 8-9 GB.