I see your point, but convnets work also on not natural data. Once you have the spectrogram, that’s a static image and if you have enough data (a representative amount for your task), you can use them. RNNs are for classifying sequences, for instance you feed them with every sample of your pcm signal and get as output a single label, or a label for every sample. Depending on the task, there are different kind of architectures. You may want to check the wavenet paper, there should be a nice pytorch implementation.