Moreover, probably you want to learn some translation invariant features, specially if you are dealing with classification. In this case, it would be useful to try to stack max-pooling layers after the convolutions. It will reduce the nb of parameters as well.