You can calculate the output size of each layer and see that when it reaches the last conv layer, the feature map gets smaller than the receptive field (5x5).
1x3x32x32 --> 1x3x28x28 #after conv1
1x3x28x28 --> 1x3x13x13 #after maxpool1 (use padding=1, if you want 14x14)

1x3x13x13 --> 1x3x9x9 # after conv2
1x3x9x9   --> 1x3x4x4 # after maxpool2

1x3x4xx   --> ?? (not enough size to perform convolution)