Hey, folks,
Wonder what pytorch’s expected behavior is when data partition is not balanced, then as a result each training process has a drastically different number of batches.   As a concrete example,  two training machines, each does 10 epoches on its local partition of data, then b/c data is not balanced between partitions, one partition has 100  batches and the other partition has 10 batches, how to make pytorch work in this scenario? Can we have the remaining 90 batches proceed without doing all-reduce?
One idea is probably to let each partition has different batch_size, but for some reason, it is hard to have that information as well.