If tensor_current is attached to the computation graph (i.e. if its .grad_fn attribute returns a valid function) then all computation graphs will be stored with tensor_current in tensor_for_accumulate. Pushing tensor_current to the CPU doesn’t change anything if the operations were performed on the GPU beforehand, since all intermediate will still be on the GPU.
I’m not familiar with your use case and don’t know if you want to store the computation graph. If not, detach() the tensor before accumulating it.