This increase in memory usage is often due to (accidentally) storing some tensors, which are attached to the computation graph, and thus the complete graph with it.
Could you check, if you are adding the model output, loss, or any other tensor with a valid .grad_fn to a list or any other container in each training step?
If so, use tensor.detach() or tensor.item() instead when appending it to the container.