I’m training a model that splits the layers to separate GPUs via model parallel. “cuda:0” starts at around 600MB and slowly keeps etching up until it causes a cuda memory error(12GB per GPU). The other GPU stays under 600MB.
I’ve tried sys.getsizeof(object), which doesn’t do much for tensors. I also now am running empty_cache and gc.collect. Still getting this issue.
Is there a method I can use to troubleshoot where the memory slippage is?