Thanks for answering. I have a few training instances that are <1 second duration. You think they might cause a problem?

That could be it. You should check the contents of short training instances and filter out ones that might be too short to contain any valid speech.