I think it might be similar to this:
https://discuss.pytorch.org/t/tracking-down-a-suspected-memory-leak/ 157
I so far found that the torch precompiled pip version worked better than my self-compiled one, but I have not found out yet why.