Solved by belzebubukas in post #2 
              
              
                I think I sorted it - the model I was using had a dropout layer, which would occasionally make it so that all the utterances of a single speaker would get dropped, causing the error. 
Sorry I didn’t provide the model, which ended up being the cause (well, when combined with GE2E). Hope this helps so…