I am interested in testing different kinds of architectures during training. For example, architecture 1 will have a specific architecture (conv-conv-pool) and then for the next epoch, I will modify it a little bit to become architecture 1 (conv-conv-pool-conv). This will go on until the the number of blocks is about 15 .
My problem is that I noticed that the memory goes up when I replace the model with a modified model. I checked the GPU memory allocation and it seems that the memory accepts the model as a new set of data instead of just replacing the old one. It wouldn’t be a problem if I am just testing one architecture at a time but I am using around 10 simultaneously.
Now I can just have batch_size = 4 instead of a faster 128 or else I will see CUDA out of memory
Please help me to release the memory by just replacing the old architecture data instead of adding on to it.
Thank you very much!!