I sense that you are talking about an idea similar to “Network in Network 2” paper. One way of implementing it can be found here: https://github.com/yangqiongyongyu/Network-In-Network-Pytorch/blob/master/models/nin.py 3 . They realize the fully connected layer in terms of 1x1 convolution layer.