Yes, I’m using LineByLineTextDataset, which already pre-tokenizes the whole file at the very beginning. The only operations that are happening before the input to GPU are the ones in the data collator - which in this case is applying dynamic masking for MLM task.
In general, should the GPU utilization be 100% while using this script?