Based on your description it seems if you set retain_graph=True each iteration will be slower.
This points to a storage of the computation graph, which would also explain the initial error message. Are additionally see a growth in memory usage?
Could you check, if you store any output of the model or the loss in e.g. a python list directly?