I have no knowledge of the internals whatsoever, but from my simple testing it seems as though it doesn’t matter, i.e. you can set require_grad to false, but it doesn’t give you any performance speed-up.