We need to not skip special tokens for CTC (wav2vec2 in particular) because of the [PAD] token.
HELLO can only be transcribed because the CTC tokens are H, E, L, PAD, L, L, L, O, O for instance.
It seems here that maybe <s> got confused with <pad> and hence is not properly skipped during decoding. Could that be it ? Also I am unsure to know how it would behave on the unicode scripts you are using (I hope if we remove <s> by properly using the <pad> token everything will work magically, but I can’t be sure.