If you see increasing memory usage, you might accidentally store some tensors with the an attached computation graph. E.g. if you store the loss for printing or debugging purposes, you should save loss.item() instead.
This issue won’t be solved, if you clear the cache repeatedly. As I said this might just trigger unnecessary allocations which will take some time thus potentially slowing down your code.