Yes, you should be able to. Here 5 are some references on fine-tuning wav2vec on common voice dataset. With a csv file having one column for location of audio clips and the other column the label, you can fine-tune it for your own audio files.