Are you measuring epoch time or total runtime? I have no direct experience with tensorflow but I remember that the set up of the graph might take quite a bit file on TF. you should probably only time the steps themselves and not the set up.