You probably need to train a bit longer. Did you train the full 30 epochs? Or did you stop at 2?
The wav2vec2 embeddings only learn the representations of speech, it does not know how to output characters. The finetuning stage learns to use the embeddings to output characters.
The usual finetuning training behavior looks sth like:

Beginning: Output random chars
Early: Output nothing - empty strings - looks like you are here?
After a while: Starts to spit out more relevant chars