The Dataset looks good and apparently the custom collate_fn doesn’t change the slow down.
You could install PIL-SIMD to speed up the image transformations.
PIL-SIMD is a drop-in replacement for PIL and you don’t have to change any calls, so this might be worth a try.