At the moment I have the following psuedo code:
def batch_conv(x, weight, bias=None, stride=1):
    for i in range(x.size()[0]):
        yi = F.conv_transpose2d(x[i:i+1], weight=weight[i], bias=bias[i,:weight.size(2)], padding=1, stride=int(1/stride), output_padding=1, groups=groups)
        y = concat(y, yi)
    return y

class AdaptiveConv2d(nn.Module):
    def __init__(self, *args, **kwargs):
        super().__init__()

    def forward(self, input, weight=None, bias=None, stride=1):
        return batch_conv(input, weight, bias, stride)


However this doesn’t distribute properly and my assumption is that the data parallel isn’t able to handle the for loop in my code.
I haven’t tried implementing a conv layer that takes a batch of weights instead of a single sample. Since I have only switched to pytorch recently and I am a bit out of depth with this.