I know when you run the first batch, it’s possible to be very slow. However, I met the following situation:
Read data: 1.64876580238
iter 60021 (epoch 8), train_loss = 2.436, time/batch = 0.396
Read data: 0.0616140365601
iter 60022 (epoch 8), train_loss = 2.617, time/batch = 0.098
When I spend a long time reading data, the time of network is also higher. The time is counted after synchronized. My dataloader includes multi-threaded loading.
Anyone has any idea?