I solved the problem. The problem was because the dataset was listed as only having a length of the same size as the batch size. And it seems the DataLoader will “hand out” remaining dataset indexes left to be processed to its workers. When the batch size is the dataset length, then only 1 process will be given indexes to preprocess.
I had done this because only a small patch of each full image was used during processing, and this patch was randomly chosen from each image during the transform part of the preprocessing. Since only a tiny part of each image was used (and the total number of full size images were less than the desired batch size), it didn’t make sense to use the actual length of the dataset as the length listed as the length attribute of the dataset. In the end, to utilize each worker of the DataLoader, I need to set the dataset length to be num_workers * batch_size. This way, the DataLoader can give each worker batch_size number of indexes to work on.