Each Data Loader pairs with one DDP instance. So if you define a batch size of 64 in your data loader, each replica trains on a size-64 batch. When training on 2 GPUs, this essentially means you train on 64*2=128 samples in one iteration. Each replica performs the forward pass independently on their separate batches, then in the backward pass, they communicate gradients computed with each other and take the average of them, so each replica has gradients as if they trained on an entire 128-sample batch themselves. Finally, each replica performs the optimization step using the averaged gradients, so they end up with the same model weights at the end of each iteration.
In essence, the speed-up comes from the fact that with n GPUs, you are able to train on n times as much data in each iteration. For this reason, the memory load on a single-node model and a distributed model are essentially the same (there may be small overheads for synchronizing the gradients but these are unlikely to influence which model/hyperparameters are chosen).