Hi Marvin,
I think a lot of the mathematics are coded up already in
https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/conv.py 52
in particular, the sizes should be there.
The main things to do are probably

figure out how to optionally call forward with _cudnn_info  instead of all the details, and apply that to grad_output in _grad_input and _grad_output instead. (Or make that a separate class and give back the info for the second derivative? I have not actually looked at it enough to be able to get it out.)
do the usual @staticmethod and self → ctx business.

In terms of arguments I think

the derivative of output w.r.t. input is a ConvND with transpose “flipped”, stride same,
the derivative of output w.r.t. filter is the ConvND of input with grad_output, dilation is the stride of the forward (based on the intuition that we need adjoints, maybe the other way round, too),
the derivative of output w.r.t. bias should be something like it is for linear.

The definition of transpose seems to be so that you don’t need to reverse the filter or things like that.
Best regards
Thomas