Hi,
Your network seems memory greedy.
You have several upscaling x5 with many many channels (1024-2048)
In case you do really need such a scaling factor (actually x(552.5)) you should think of reducing the amount of channels.
Going from 1024 to 2048 means that you are gonna have 1024 * 2048 * H*W  float numbers + the mem required for backprop.