Hi @albanD,
Thanks for the answer ^^. You are correct, the register_hook works without retain_grad(). The code works perfectly fine during the eval() phase now.
I was confused about using if score.requires_grad: on the score variable as it is not a leaf-node variable, but an intermediate-node variable.