It was a problem with the sampling rate in my case. I was mistakenly using too short segments of audio so the model was unable to learn anything. You might as well try cleaning your audio if it’s too noisy.