Just tried loading entire dataset onto GPU, the speed gain is only about 0.08s per iteration. It is still very much slower than mxnet for the larger network below (mxnet’s 1s vs pytorch’s 4.27s).
Seems like larger network is even worse (the new one above).