Both can improve or reduce the performance.
As described before, the model performance depends on the general architecture and you, as the researcher, would have to run experiments to see which kernel sizes, number or kernels, and other hyperparamters work well for your use case.
If there would be a general rule such as: the larger the filter, the better the model performance, one could simply use the max. size and would get the best model (which is of course not the case).
I would recommend to start with an online course such as FastAI 2 or any other course which gives a good introduction to ML/DL.