After reading your comment I was thinking what would happen if you were to blow up the network? You could try depth, width and then both. In other words try to heavily overparameterise the network and see what happens?