Thank you for your help.
Can cudnn also effect the performance?
I didnot record the exact time, but the code that used to take around 7hrs (on TITAN Xp) to finish is taking more 11 hr (rtx 2080 ti) now.
oh and also we mistakenly didnot have cudnn on the first machine at all, but we have it on the second machine (same verdsion of Cuda 9.0) same version of pytorch. now im gonna go ahead and disable it to see if any changes will happen or not