I trained a network, but I find the memory it occupies in GPU enlarges every epoch.
In my program, every epoch includes validate and train procedure. It first validates and then trains.
When the program runs, the memory it occupies enlarges by epoch until the GPU-util is all 100%. It stop to preceed, but the program is still on with memory occupied in GPUs.
How does it go?
Can anyone helps?
Thank u very much!