Since your input shape is [batch_size, 3, 21, 21], the flattened activation before the linear layer will have the shape [batch_size, 16, 2, 2], so you should change the input features to 16*2*2 as well as the flattening operation (or use x = x.view(x.size(0), -1) instead).