You are wrapping the kernel into a new nn.Parameter so that all values will get independent gradients and will thus diverge.
To use a and b as the only parameters, you could create the filter using torch.cat and torch.stack operations and use it in the functional API via F.conv2d(x, kernel).