Thanks a lot @ptrblck_de for your help throughout the discussion.
Here’s an enigma. Although GPU Tesla P100 provide better times for data size of 1000, the training time increases rapidly by just increasing the data size, as corresponding times are mentioned below:
1k  :  0.117 s/epoch
11k :  0.327 s/epoch
21k :  0.546 s/epoch
31k :  0.762 s/epoch
41k :  0.981 s/epoch
51k :  1.206 s/epoch

Do you understand why there’s so much difference in time increase for two different GPUs?
Also i tried a code from a research paper written with tensorflow. Having made no changes to it, training time on Tesla P100 was 6.3 min while it’s mentioned in the paper that NVIDIA Titan X did it in 1 min. I can’t figure any reason since i think P100 is more powerful.