Yes, this is how it’s supposed to work. However, for any useful network, full-Hessian is going to be too large to fit in memory. Block-diagonal is a common approximation because off-diagonal blocks may approach rank-1 matrices for deep network, hence not too useful.
You can save some time by computing block Hessians for all layers from same set of backward values, here’s an example – https://gist.github.com/yaroslavvb/6b08eb8f683b646785d39e97679fdb2e 308