It takes me 10 days and did not find the solution to this problem. It turns out to be the “shuffle=True” cause the problem, I tried several ways
(1) using data prefetcher
(2) creating the dataset using h5py dataset format, which is also too slow.
(3) I tried to create a dataloader before each epoch and using random index when creating the dataloader. But it still did not work.
(4)  I use the random index as index to h5py file
np.random.seed(i) 
np.random.shuffle(train_index)
ds = FaceLoader(H5Data, train_index)
trainloader = DataLoader(ds,
                batch_size=args.train_batch_size,
                shuffle=False,
                num_workers=args.num_workers, 
                pin_memory=True,
                drop_last=True,
                persistent_workers=True)

The training is okay on 1/4 of dataset(320K images) but getting slower and slower on whole dataset (12000K)
My CPU has 4 cores and 8 processors, My GPU is nvidia-smi
If someone has the same issue and solved it, please let me know how you solve it.
Any help or suggesstion would be appreciated.