Maybe it is the reason:
When the folder contains images whose names are with  specific meaningful prefixes, then the dataloader load the images sorted alphabetically.
Take two classes dog and cat as an example, given cat label 1 and dog label 0.  Suppose 1000 images are dog_001.jpg, dog_002.jpg, etc,  200 images are cat_001.jpg, … Then you get labels with all 1s.
Because images with the prefix of cat should be returned first by alphabet.  So this explains your case.