In fact, @PiotrSokol was kind enough to share his implementation:

  
    
    
    [Adding functionality] Hessian and Fisher Information vector products autograd
  
  
    Hi, 
I implemented Hessian and Fisher Information matrix (FIM) vector products and was wondering if there’d be interest in adding this functionality. The FIM products are optimized, in the sense that they analytically compute the Hessian matrix of the logarithm of the loss wrt. the output layer for a set of commonly used losses. 
Best, 
P. 
Edit: Tagging you in 
@smth, @apaszke 
Editing for posterity since no one wants to discuss how to submit this PR. 
import numpy as np
import torch

def _chec…
  


Best regards
Thomas