Solved by rvarm1 in post #2 
              
              
                DataParallel can be applied to any nn.Module. For the first example, you are correct in that every nn.Module (i.e. the blocks) will be data parallel as well, since the entire module is. 
The second example mostly seeks to show that you can have a regular nn.Module, but have a data parallel module co…