The graph ends up taking 2/3s or more of the total memory during training. Some optimizers more and some less. I’d be surprised to see the outputs larger than the model and the gradients. Are you storing all of the outputs during training?
Do you have a second GPU? You may want to clock the time it takes for each part in the process to ensure you are fine with any latency overhead.
Without knowing more about what you’re trying to accomplish, that is all I have for suggestions.