agt:

What if instead of N 3x3 convolutional layers, I applied the same 3x3 convolutional layer to its own output N times?


Each convolution (or a set of filters) would learn different features depending on where they are placed in the network. So, the convolutions early in the network would learn to identify lower level features (such as lines and points) while the later convolutions would learn to identify higher-level features (such as eyes and ears). If you simply reuse the same convolution N times, the parameters would be shared. Hence, it would hard (or impossible) for the convolution to clearly identify (or learn) the different features in your input image.



 agt:

Should I somehow use a hidden state between the applications like in an RNN?


Remember, RNNs are based on the concept of sequence and BPTT. CNNs are not based on that idea, hence there’s no hidden state shared between CNNs. Hence simply using multiple CNN layers is the best approach.