First of all: if your GPUs are on one machine there is no need for DistributedDataParallel just use the plain DataParallel instead.
Second: Cuda-calls are asynchronous. If you split the network this way, the one to be trained would (probably) be slower and therefore the GPUs with the already trained net will be idle. To optimally use your GPUs, I would recommend to wrap both models in an abstract module and distribute this wrapper with DataParallel on all 4 GPUs with an increased batchsize. In this case a copy of every net will be created on each GPU to split the batch across your GPUs but the gradients will be accumulated.