Thanks for the reply!
Now I know why the timing is weird.
For the slow part, I time the training on one batch and print it in the code. It shows that the training speed is around 13 instances per sec. With larger graph using tensorflow, the same machine can easily achieve thousands of instances per sec.
In the pytorch code above, every batch takes about 8 sec.
Thanks.