A more specific problem I’m having doing this is that when I use a Conv function inside of forward or backward function, such as
class TestConv(torch.autograd.Function):
def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1):
    self.ConvFct = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding, dilation, groups)
def forward(self, input):
    output = self.ConvFct(Variable(input))
    return output
def backward(self, grad_output):
    return grad_output

I get a RuntimeError: _Map_base::at. If I could use conv functions this way I could just implement the backprop rule myself.