An increasing memory usage usually points to some code which stores tensors, which are still attached to the computation graph and would thus not only store the tensor but the entire graph with it.
Could you check, if you are appending the output of the model or the loss to e.g. a list without calling detach() on them?