Could you just blow up the units and check it again?
Just multiply the number of hidden units with 10 or 100 and check it again until you run into an out of memory error. If nvidia-smi still shows a low memory usage, something should be wrong with it.