I found that in  each epoch during pytorch training, dataloader for loop takes almost half of the training time. So I set num_workers=4 in dataloader, now each epoch takes 1.44s, dataloader loop takes 0.8s, so  training time is 0.64s comparable to Keras.  Pytorch cpu usage is normal when I limit the cpu core nums.
torch.set_num_threads(8)

I don’t know how keras makes its data pipeline so fast. Still,  Keras gets lower loss.