It depends a bit on your image stats etc.
Usually a kernel size of 3 works quite good, as a lot of models use it (see vgg etc.), for a “natural” image of approx. 224x224.
If you have a medical image (e.g. MRI) in a high resolution, the kernel size might not be the best.
A good resource is Stanford’s CS231n 41 to get a feeling on convolutions.