lichao:

but should reasonably call “cudaStreamSynchronize(stream)” in a suitable place


That is correct and synchronizations should only be called, if necessary.



 lichao:

the memory allocated for cuda in advance, such as calling cudaFree, but cuda is not calculated at this time, the release of these cuda resources is triggered by the release of the temporary Tensor in the network (if I am wrong please correct me, thank you), this will cause serious problems, right? So I want to know how the pytorch guarantees that cuda is already calculated when the temporary Tensor is released, I think that “cudaStreamSynchronize” should be called once before the intermediate Tensor is released.


If I’m not mistaken, the synchronization happens here 48.
Are you seeing any issues with using CUDA or is this more of a general question?