It&#x27;s not about the design of the network,  it&#x27;s about the training.  Optimizing the weights of a DNN is a non convex optimization problem solved using gradient based search for weights that minimize a loss function.  If you can only search in 10 dimensions,  it&#x27;s harder to move in the correct direction.  Once you&#x27;re searching in 1000 directions,  you can usually find a path that gets you to the right place.  Think of it this way: there may exist a 10 dimensional solution,  but it requires learning a very dense solution.