In order to deal with high resolution 2K images and progressive training of GAN and limited GPU memory(12GB), I am forced to split my model on 4 GPU. However, there seems to be the problem of CUDA OUT OF MEMORY error each time.
And the strange thing is that it happens after one iteration(which means forward and backward pass are both performed). I have checked my code and there definitely will not be any case that my code behaves in the following way:
total loss = 0
for (iteration ):
loss = …
total loss = loss + total loss
Except the above case, I really cannot think of a possibility how there will be an CUDA out of memory bug after one iteration (Also, I do not accumulate gradient as well)
Can anyone help?
My memory usage after the first iteration
memory.used [MiB], memory.free [MiB]
4229 MiB, 7978 MiB
3686 MiB, 8521 MiB
1104 MiB, 11103 MiB
5754 MiB, 6453 MiB
Best Regards