What I’d do it to see how one simple model does that like this one: Load and preprocess images  |  TensorFlow Core 6
maybe structure the folders to have files from a specific class into a dir with that name, like the flowers dataset does, and then use the same strategy