I’m not sure I understood it properly. but one solution would be to adapt the current implementation of nn.DataParallel so that you can better control when you gather the parameters.
For example, have a look at those lines 122. I believe you can adapt it somewhat easily so that you only replicate the model and gather the results from time to time.