You should still be able to freeze the submodules via model.module.a after you’ve wrapped them in nn.DataParallel. As long as your changes are done outside of the forward pass, which is executed on each replica, it should be fine.