Why deep learning works even though it shouldn’t
> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all.<p>Can someone who knows more about DL than I do help me understand this a little better?<p>The article uses the analogy of walls:<p>> Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to be surrounded by walls. In 4 dimensions, you can walk through walls. GPT3 has 175 billion parameters. In 175 billion dimensions, walls are so far beneath your notice that if you observe them at all it is like God looking down upon individual protons.<p>I'm struggling to understand what this really means in 4+ dimensions. But when I try to envision it going from 1 or 2 to 3 dimensions, it doesn't seem obvious at all that a 3D space should have fewer local optima than a 2D space.<p>In fact, having a "universal function" like a deep network seems like it should have more local optima. What am I missing?