Well ijt’s difficult to say.
I would use Binary Cross entropy rather than cross entropy as you have a binary case.
I also would do a pooling in after the CNN. Think that those features are spatial and you don’t really care about where the speech is located in the original mel. It also forces each neuron in the linear layer to distinguish between speech/non-speech.
Rather than taking a subset, take a single batch, load it in the gpu and backprop with a simple clean script. This can help you to debug whether there is a bad network desing or a bug in the code.
If everything is okay then I would start from checking the dataset and making sure it works as supposed. If it does you can go for a simpler problem. For example, two clean different categories. Using enhanced speech where the human voice is clearly predominant. In short you have to be sure the task is suitable.