My bad on the previous reply. You only need to set eval mode. No need to make a separate affine transform.
The branching to call training/eval backward is at https://github.com/pytorch/pytorch/blob/04ad23252a3ce592c5e5c30c6fd87000f8d178cf/tools/autograd/derivatives.yaml#L1038 404 (if you have cudnn installed)