What does the 1/11 signify?
How many iterations through your training loop does it take to exhaust the data loader?
Keep in mind that the data loader is going to assign an entire batch to each worker, and with 32 workers you’re going to be hammering the disk pretty hard.  Once you have the first batch ready to go, the next 31 should also finish right around the same time. That means that batch two (through 32) should be ready with zero latency, and the first loader process should be working on the 33rd, which is why everything but the first batch is quick.
You can try having two data loaders active at the same time, so that when you get to epoch 2, the second DL already has everything loaded up and ready to go. Something like:
this_dl = DataLoader(..., num_workers=16)
next_dl = DataLoader(..., num_workers=16)
this_dl_iter = iter(this_dl)
next_dl_iter = iter(next_dl)

for epoch in range(epochs):
    for batch in this_dl_iter:
        # ...training, etc...
        this_dl_iter = next_dl_iter
        this_dl, next_dl = next_dl, this_dl
        next_dl_iter = iter(next_dl)        

Note: I’m assuming without checking that trying to iterate over the same DL twice at the same time won’t work. Double check that.