I guess it depends on the size of this subset and how well this subset reflects the data distribution of the complete dataset. If some subsets contain a highly class imbalance, you will fit your model on these majority classes, which most likely will hurt your overall performance.
If it’s possible, I would rather lazily load the data (using Dataset and DataLoader) and use the complete dataset.