I am trying to implement a more general version of grouped convolutions that takes a list split whose sum is equal to the number of input channels channels and internally groups the convolutions according to this partition. This is in constrast to PyTorch’s grouped convolutions, which only allows groups of the same size. Here is my attempt (the important part is the forward, please ignore the references to DilatedCNN in the __init__) :
class SplitDilatedCNN(nn.Module):
    def __init__(self,channels=4,depth=5,kernel=5,split=[3,1]):
        super(SplitDilatedCNN, self).__init__()

        self.channels = channels
        self.depth = depth
        self.kernel = kernel
        self.split = split
        self.channelsAux = 0

        self.dcnn = nn.ModuleList()
        for i in range(len(split)):
            self.dcnn.append(DilatedCNN(split[i],split[i],self.channelsAux,self.depth,self.channels,self.kernel))

    def forward(self, x):
        x = x.split(self.split,1)
        x = [self.dcnn[j](x[j]) for j in range(len(self.split))]
        return torch.cat(x,1)

The forward pass seems to work, but in the backward pass I get an

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

and have absolutely no idea why. I can’t see any in-place operations!?