I’m just a begginer, but i have gone through the Google’s speech recognition API. What they do is record 3-5 seconds of background noise, and then use the intensity of recorded background noise as base level. Any substantial change in this intensity triggers its recording and the pre-trained speech recognition determines wether the sound is a spoken word(say the wake word) or just a random sound.
Coming to the your CNN implementation , it really depends on one question . Can you differentiate between human speech and no human speech pictures? if yes then the CNN ideally would work. Also does the spectrogram show difference between noise and human speech?
The implementation , as Juan said above has only 1 minor issue. Use BinaryCrossEntropy for binary classification. I doubt that not using maxpool matters much , however it is something you should play around with. Take inspiration from other successful NN for the task. You can also do  transfer learning , however this might be an overkill.