@Sai_Gayatri_Prasad  what is your application?
For network with ReLU activations, FIM is the same as the Hessian and the Gauss-Newton matrix. For general activations, it’s the same as the Gauss-Newton matrix. See Appendix C.2 in https://arxiv.org/pdf/1706.03662.pdf 32
GN matrix is much cheaper to compute than generic Hessian. For instance, getting diagonal of GN has similar cost to getting the gradient, while diagonal of general Hessian is likely intractable (see Section 6 of http://www.cs.utoronto.ca/~ilya/pubs/2012/CurvProp.pdf 14)
For neural network with k outputs, you can compute GN matrix with k calls to .backward(). The main tricky thing is figuring out the thing to feed into “.backward” which depends on the loss function used.
Full GN matrix will need >TBs of space even for tiny MNIST networks, so in practice you’ll need to take approximations like diagonal, kfac, etc. An example of doing this for cross-entropy loss is here: https://github.com/cybertronai/autograd-lib 18