Solved by ptrblck in post #2 
              
              
                If tensor_current is attached to the computation graph (i.e. if its .grad_fn attribute returns a valid function) then all computation graphs will be stored with tensor_current in tensor_for_accumulate. Pushing tensor_current to the CPU doesn’t change anything if the operations were performed on the …