One possible answer is to take a mean after the gradient updates like so
kernel.weight.data = conv.weight.data.mean(0).unsqueeze(0)

Is this the best way to do it. Or is this even right in the first place?