I ran my code on another GPU server and each epoch now consistently takes around 680 seconds which is significantly less than the previous run. I guess there is something wrong with my environment.
Any suggestions on what might be the possible issues or how to make the above implementation faster are welcome.