Hi,
There is not much you can do to reduce GPU memory usage except reducing the batch size and use inplace operations for activations like ReLUs if possible (if you see an error saying that a tensor needed for backward has been changed by an inplace operation, then that means that you cannot use them at that particular place).