Solved by ande765a in post #3 
              
              
                Thanks! - I think that was the problem! 
After changing the model from operating directly on the waveform to instead operate on the MFCC it worked! I think the input lengths when operating on the waveform were just too large compared to the target lengths for the CTC to work properly.