A Pythonic way to do a data parallel on a sequence of modules is to group them in a container, and use data parallel on that container. You could just remove the data parallel code from features. And if you really need to do it differently, we have gather, scatter, replicate and parallel_apply inside torch.nn.parallel. Just keep in mind that they’re not documented right now and they still might change.