Solved by ptrblck in post #8 
              
              
                You could pass an input with a larger spatial size (height and width) or alternatively remove some pooling layers or other layers, such as strided convs, which would reduce the spatial size too much.