I have a Sampler to do backpropagation through time, similar to the one in the torchnlp examples. This requires giving the batch size of training as an argument so that batches are properly continuous (e.g. if the dataset is [abcdefghi] and batch size is 3, the batches are [adf] [beh] [cfi]).
My question is, how does this work in a distributed setting? From reading the code of the distributed sampler, I got the impression that each process gets its copy of the sampler. In this case, I would need to know the local batch size in every process to properly separate batches. Is there a general rule to determine batch size per process (like an equal portion per GPU?), and if not, how could one determine the local batch size?