Are you using the linebylinedataset or not? The only thing I can think of is that you have an IO bottleneck; the GPUs process data faster than that the data can be read and tokenized on the fly. You can try profiling your script to see where the issue lies.