If you want to use different batch sizes for the datasets, you could either use different DataLoaders or probably create a custom sampler, which would use the predefined sample indices given your batch sizes.