Does that also happens with Tensors directly created by PyTorch (with zeros or ones for example) ?
Are GPU tensors also impacted?
Do you have a small code snippet we can use?
Can you check in top/htop if there is high memory/CPU usage at the same time.