The work (in terms of CPU time) is not the problem - it’s rather that a complete hessian for a 400x4x5 conv layer needs several terabytes of memory.
Discarding is fine (although inefficient), so I will try to go with that for now.