Solved by tom in post #8 
              
              
                So the immediate takeaway from the above discussion is 

replace time.time with time.perf_counter()
have a torch.cuda.synchronize() before taking the the start_time,
maybe don’t take the first batch (of a given size)