we are working on enabling this. This will be in the next major pytorch release 2 months away. It wont be a full 2nd derivative but it will be the Hessian-vector product (whether it’s tf.gradients or in pytorch). We are tracking this feature as double-backprop / double backwards.