Solved by J_Johnson in post #2 
              
              
                I found some tools here: pytorch.org/docs/stable/cuda.html 
under “Memory management”. 
Figured out my problem was that I was using the outputs for some statistical calculations and forgot to put them back to cpu before performing those. Somehow that was causing the issue of tensors being duplicated…