Hey,
I have a network which overrides the parameters() function to only include trainable parameters. This has worked well until I tried to run it with DataParallel. I guess I was not supposed to override it because DataParallel does not work with my model. Here’s an example:
# Python 3.6
# Pytorch 4.1 installed via anaconda
import torch
from torch import nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.module_list = nn.ModuleList()
        self.module_list.append(nn.Linear(10, 10))
        self.module_list.append(nn.Linear(10, 10))
    
    def parameters(self, only_trainable=True):
        for param in self.module_list.parameters():
            if only_trainable and not param.requires_grad:
                continue
            yield param

net = nn.DataParallel(Net().cuda())
net(torch.rand(1, 10))

This throws a NotImplementedError. If i set requires_grad=False on a module I instead get a KeyError in torch/nn/parallel/replicate.py
The solution is easy, just rename the function to something like trainable_parameters().
However, I’m a bit curious, should parameters() never be overridden? It worked perfectly fine when running on single GPU but I guess the function is used internally in some other parts of Pytorch? Or did I just not use yield properly?
Thanks in advance