There is one weird phenomenon that after I doubled my batchsize in a training process, the data loading time is almost doubled ( about 1.8 times long as original).
I measure the data loading time by measure the interval between the end of each iter and beginning of next iter. ( That is when the for loop to fetch next batch).
I simplified the logic as follows:
import ... 

end = time.time() 
For (input , label) in loader:
data_load_time = time.time() - end
output = model(input)
loss = crit(output, label) 
...
end = time.time() 

I conducted experiments on a single gpu to compare bch32 and bch64 settings:

gpu 1 batch 32
Time 2.505s (2.551s)	Speed 12.8 samples/s	Data 1.808s (1.849s)
gpu 1 bch 64  (cpu 5,  worker 24)
Time 4.707s (4.897s)    Speed 13.6 samples/s  Data 3.453s (3.599s)

It shows that with larger batchsize, the dataLoading time is nearly doubled. (around 1.8).
With multiple-workers, the loading process should be conducted before hand and will be ready when it is needed.
So with enough workers, larger batch should not delay much comparing to a small batch setting as the loading is handled parallelly with multiple cpu thread.
This causes troubles.  It means more gpu resource (either more gpus or larger G-mem) will not help to reduce the training time, as the data loading time is proportional to batchsize. Even you get enough GPU to handle more data at same time, the dataloading will delay everything up.