Couple of hints.
Despite you use fp16 gradients are still computed as fp32.
fp16 is supposed to save MAC ops but you won’t gain much memory. In addition, not all the gpus were optimized to work with fp16. It helps in gen 30XX and quaddros, but not in 10XX and don’t remember in gen 20XX. Check yours.
Another core question is how you coded down and up.

Reducing the filter size is critical to reduce the memory.
Replacing upconvolution by interpolation
Removing maxpool in favor of a convolution with stride
Use in-place relu
IMO there is not much more u can do.