Solved by jshanna100 in post #5 
              
              
                I tracked the problem to cudnn.benchmark. Only the first iteration takes very long, but if you wait for that to pass, the subsequent iterations take the correct amount of time.