From LeNet to EfficientNet: The evolution of CNNs
VGG: The power of 3x3 convolutions

While all previous iterations of CNN models believed in the idea of bigger receptive fields (for example, AlexNet had 11*11 convolution kernels), VGG proposed the idea of breaking all of these down to just 3*3 convolutions. According to the VGG architecture, multiple 3*3 convolutions stacked together are capable of replicating bigger receptive fields and with more nonlinearities present in between them (in terms of activation functions), it can even perform better than their counterparts with bigger receptive fields. They even introduced 1*1 convolutions in order to further increase the nonlinearity present in the model. VGG models have become quite famous since then and are used even today in various tutorials to introduce newcomers to CNNs.

VGG-16 architecture [3]

ResNet: Tackling vanishing gradients

The general trend of simply stacking multiple CNN layers to create deeper models soon came to a halt due to a very common problem in deep learning, known as the vanishing gradient. In simpler terms, while training a CNN the gradient starts from the last layer and needs to travel through every layer in the middle before it reaches the initial layers. This can cause the gradient to completely vanish somewhere along the way, making it difficult to train the initial layers of the model. Residual or shortcut connections were introduced by the ResNet models which created alternate paths for the gradient to skip the middle layers and directly reach the initial layers. This allowed the authors to train extremely deep models that were not performing well earlier. It has now become a common practice to have residual connections in modern CNN architectures.

ResNet architecture [4]

MobileNet & MobileNetV2: Moving towards edge-friendly models

The general trend of CNNs at this stage was creating bigger and bigger models to get better performance. While the advances in computing power provided by GPUs allowed them to do so, a new range of products was also getting noticed in the ML world, known as edge-devices. Edge-devices have extreme memory and computation constraints but have opened doors to many applications that a GPU cannot be applied to.

In order to create lighter CNN models that can perform at par with the existing state-of-the-art, MobileNet was proposed. MobileNet introduced the idea of separable convolutions (which I talk about in detail in my previous blog). In simpler terms, it breaks down a 2D convolution kernel into two separate convolutions, depthwise — which is responsible for collecting spatial information for every individual channel, and pointwise — which is responsible for interaction among various channels. Later, MobileNetV2 was also introduced with residual connections and other tweaks in the architecture to further reduce the model size.
['Prakhar Ganesh']
['Machine Learning', 'Deep Learning', 'Convolutional Network', 'Towards Data Science', 'Data Science']