In the example why divide by ngpus_per_node though? Shouldn’t we divide by world size?
And if we make the batch size smaller then how are we consuming the data faster? If we iterate over the DataLoader it will only use 64 items at a time.