Thanks for answering. I have a few training instances that are <1 second duration. You think they might cause a problem?
I also ended up trying this previous suggestion
model.config.ctc_zero_infinity = True
and it seems to be working so far.