I’m not sure how your actual prediction look like, but assuming they would have the same shape as your weight matrix, you could simply multiply both before calling loss.backward().
Let me know, if that would work!