How much memory does your GPU have? Your model might just be too big. You could then either reduce the spatial input shape (if possible), slim down the model, or use torch.utils.checkpoint to trade compute for memory.