Hi,
In my shallow view, the layers is nn.Module same to the whole network, I think if you solve the problem calling dataparallel on each layer individually, you could also calling dataparallel on the whole network and register_backward_hook to specific layers.