If you have the mask for the current pattern, you could use it to zero out the initial randomly initialized weight matrix as well as the populated gradients after the backward() call.
Each backward will accumulate the gradients in the param.grad attribute.
Let me know, if you get stuck.