Nice that it works now. The reason it works only with one specific kernel size is that when you change your kernel size, the behaviour of your conv layer also changes considering the resulting output shape. And this is key for flattening because you have to take this into account when setting up the number of nodes in your Linear layer. It also depends on your input image size. E.g. for kernel_size = 5 you get a smaller output of your conv2 layer than for kernel_size=3.
Edit: You also could change e.g. the padding of your conv layers, the decision to change the number within the linear layer is not the only one which works. The important thing is that your linear layer is expecting the correct shape.