Hey, Apologize for a delayed response. Why do you think backprop won’t happen or have issues with it?
We can always break up a bigger batch into smallers and still consider all those smallers ones as a single batch altogether. Similarly, when we have split out batch, now different I/ps are passing to different model code blocks, post forward pass, the grads will be combined and all of that will be send back together before the splitting point.
Also you night get little poor results but that depends on your BS as well,
Also you might want to look into collate_fn!