Sparsity of connections in a CNN - why is it beneficial? In his CNN video, Andrew Ng explains the benefits of convolutions over FC, and he mentions sparsity of connections as being one of them. Can someone please explain what is the benefit of sparsity when going from one layer to another?

Doing the same thing that a convolutional layer does with a fully connected (FC) layer requires a lot of zeros in the weights. So, the convolution operation is equivalent to sparsity + weight sharing in a FC layer. If the same thing is to be achieved with a FC layer, there would be a lot more parameters to learn and the network would be more prone to overfitting. You don't want to learn something with 1K parameters if you could learn it with 10 parameters instead. Then, you'd have to apply regularization techniques to fight with it. Apart from others, one of those regularization techniques is Lasso, which promotes sparsity. But the convolution operation already achieves this by design. Besides, there will be many more different solutions as the weight sharing is hard to achieve within the FC layer.
