The replicas will be recreated in each forward pass using nn.DataParallel (which is also why it’s slower than DistributedDataParallel).
Your code will most likely just use the single model, as seen here:
model = MyModel()
model = nn.DataParallel(model)
model.to('cuda:0') # push to default device

output = model(data) # DataParallel will automatically create the replicas
loss = ...
loss.backward() # DataParallel will automatically call the backward pass on all models and reduce the gradients
optimizer.step()

torch.save(model.module.state_dict(), path) # use the default model

You can find more information about the underlying workflow in DataParallel in this blog post 13.