Hi @ortho-stice
This is expected behavior. Here is the source code of DataParallel: https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/data_parallel.py#L148-L153 8
What happens is that, in every forward pass, DataParallel will

scatters the input to all GPUs
replicate the model to all GPUs
launch parallel_apply so that every GPU will run its own forward pass using its input data split in parallel.
gather all outputs to the output device

So the model replication only occurs in the forward pass, and hence you won’t see those model replicas outside the forward function.
BTW, we do recommend using DistributedDataParallel which only replicates the model once in constructor instead of in every forward invocation.