I face the same problem with you and I find the problem results from the upsampling layer.
When I use nvidia-smi to monitor my GPU memory usage, I find that during some time, the memory requirement for transpose2d for upsampling will be doubled and it tells me out of memory…
Maybe consider to reduce the batch size.