It seems like your vocabulary size does not match the vocab size from the original model. You can remove the old LM head manually and initialise a new one:
state_dict = torch.load(f"{model_args.model_name_or_path}/pytorch_model.bin", map_location='cpu')
state_dict.pop('lm_head.weight')
state_dict.pop('lm_head.bias')

model = Wav2Vec2ForCTC.from_pretrained(
        model_args.model_name_or_path,
        state_dict=state_dict,
        ...
    )