Okay, thanks for the tip!
So I have been running a simplified version of my script and now the model is able to learn. I also tried to build a model based on 1D convolutions: the “time” axis becomes channels. I was reading that it makes more sense theoretically. It seems to work OK and way faster than the conv2D (due to less parameters)
However, I have another question regarding the training of the model. My dataset is very imbalanced: 95% of no speech for 5% of speech and it seems that the model struggle to “learn” what speech look like - I get up to 60% accuracy for speech. Is there any method or architecture I should look for to improve the performance on such dataset?
Note that the dataset is artifically created: I overlapped VoxCeleb / Librispeech voices on only 5% of the ecosystem record. This percentage was decided based on the real-life occurrence of human sound in an ecosystem.
Would it be better to build a balanced dataset for the model to better “learn” what speech look like, even though this is not a realistic situation. Or is it better to focus on improving the model with the 5% occurrence of speech?
Thanks for the help!