Nothing looks blatantly wrong code-wise. I tested the custom conv module and it seems gradients are accruing and seem reasonable enough.
Is there a reason why you’re sharing the same weights for each dilation? This concept seems very strange to me. I could see the kernels being weird because they’re trying to operate locally (dilation=1) and also being told to operate more globally (dilation increasing).