@ptrblck this is a little more breakdown of what I am seeing for a training set of 1600 samples, each with length 66650 and a test set of 4000 samples with length 66650.
Torch:
Epoch: 1/2000 | Time: 4m12s (Train 2m23s,  Val 1m48s)
Tensorflow:
Epoch: 1/2000 | Time: 1m52s (Train 1m07s,  Val 0m44s)
Using my mackbook, so no gpu support. Just CPU. It seems that I am running out of time looking at every possible glitch I may have. I ran it on cluster having GPUs. The scenario is the same, where as the tensorflow model takes only 7 seconds to complete one epoch. Thus, can’t afford to have my torch model training for 5 days in the cluster.
Anyway, thanks a lot for your effort on it. Please let me know if you have any suggestion.