Many thanks! Actually, I’m think there is a bottle neck of my training process and I’m trying to find it out. I am training a 3-layers MLP model but the training is irrationally slow.
Though I did a preprocessing of data in the training loop, I think it won’t influence the training time too much, because the preprocessing should be accelerated by CUDA. 
屏幕快照 2019-08-21 下午8.32.09.png1278×482 31.8 KB
 So I think whether the data loader is working efficiently. The data loader load a np.array object in the .getitem method.
The most weird thing is the training seems be slower epoch by epoch, though I use a torch.cuda.empty_cache, it doesn’t help much.
A preload is also implement, so the GPU is not starving.
屏幕快照 2019-08-21 下午8.27.48.png1168×1278 152 KB