After a few more epochs and training the whole data, I got 32.09% WER and 8.23% CER on the test set.
Give it a try! I provided the preprocessing and the training arguments in the following link.

  
      huggingface.co
  
  
    

m3hrdadfi/wav2vec2-large-xlsr-persian · Hugging Face 9