I think I’ve figured it out. I will post it here, so that if someone else comes here with the same issue, they can perhaps be assisted by my answer.
It turns out that it was my mistake, as even though I was loading an old model, I was using a new processor (tokenizer/feature extractor). Now, the problem here is that while the model worked as it should, the decoding of the transcriptions was simply wrong, because the labels it outputs were extracted from an old vocabulary.
When I tried to decode the outputs with a new processor (who was build from a different vocab), the matchings were simply wrong.