Do the images show the stride pattern of each conv kernel or are they kernel layouts with valid weights?
In the latter case, you could zero out all invalid weight positions at the initialization as well as their gradients after each backward call.