@bsridatta I verified that my data was being replicated across batches by actually printing them out.  I’m not sure, but this problem may be a product of using pytorch-lightning, which makes a copy of the dataloader for each GPU.
In any case, I was able to fix the problem by creating an array of pointers to the start of each training example in my file using an approach similar to the one used here 79.  This allowed me to quickly sample random training contexts from my text file without needing to read it into memory.