I havent tested this out in pytorch itself, but generally speaking any time you ask the gpu to do something, it’s fairly asynchronous. So, you send a request, in the mail as it were, and at some point in the future, when the gpu feels like it, has finished its breakfast, reading the morning paper etc, you get the results back.
Now, ok, it doesnt really read the paper etc, but it does take a while to do stuff. If you send more stuff whilst the first stuff hasnt finished, obviously the second set of stuff will be delayed for a while.
copying data to the gpu counts as ‘doing stuff’.  it may look instantaneous, because the instruction returns immediatley, but it’ll take a while.
Hunt for an instruction with a name like ‘sync’, or similar, and have a play with that. in fact there’s an example I made earlier here:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Reductions returning scalars cause implicit sync-point 39
    

    
      
        opened 
        
          
        
        Jul 6, 2017
      
      

        
          closed 
        
          
        
        Jul 7, 2017
      
        

      
        
          
          hughperkins
        
      
    
  



  Scalar reduction results are currently being returned as host-side scalars, causing an implicit sync-point, and gpu->device copy. See the following results:
=================
it...