Would it be possible to reset the particular part of your weight matrix after each iteration?
If so, you could wrap your code in a with torch.no_grad() statement and add if after optimizer.step() was performed.