Could you try to narrow down a minimal code snippet and post it with random input tensors so that we could try to reproduce this issue?
Also, in case you are seeing an increased memory usage besides the slowdown, I guess you are storing the computation graph (accidentally) in each iteration.
You can also post code snippets by wrapping them into three backticks ```, which makes debugging easier.