at batch size of 7, it’s taking 5GB for me.
 5675MiB / 16273MiB

You wrote the network pretty efficiently, i dont think there’s memory improvements possible.
Try adding this flag for speed improvments:
torch.backends.cudnn.benchmark=True