If you are not storing the loss directly in e.g. a list or any other tensor, which is attached to the computation graph, your model might just use too much memory.
Are you seeing the OOM issue in the first iteration(s) or later in training?
In the former case, you could try to trade compute for memory via torch.utils.checkpoint, while the latter case points towards storing tensors without detaching them.