Thanks, I find the bug is that I want to use different streams to overlap computation and data transferring in my model definition but stream will run in an uncontrolled way actually. I know tensorflow will put computation and data transferring in different streams by default to try to overlap. But it seems not easy to do this overlapping in Pytorch.
Anyway, Thanks for your response.