@ptrblck , thank you! I tried to add torch.cuda.synchronize() at the end of each for loop. Indeed, it works! The time of each model forward now is consistently ~0.0015s. The only problem is, torch.cuda.synchronize() itself takes ~0.27s, so the complete testing takes nearly the same total time as before. I guess this points out where is the problem. I also note this issue was reported before. Now I am wondering, if this can be resolved somehow. Any clue?
Thank you ALL!