@mckowski any reason you dont want to make gram a Variable? That would then fix the issue I think?
(Edit: by the way, if you have requires_grad=False, it wont backprop through gram, in case this was a concern?)