Evaluation of Few-Shot Learning for Classification Tasks in the Polish Language
<|reference_start|>Evaluation of Few-Shot Learning for Classification Tasks in the Polish Language: We introduce a few-shot benchmark consisting of 7 different classification tasks native to the Polish language. We conducted an empirical comparison with 0 and 16 shots between fine-tuning, linear probing, SetFit, and in-context learning (ICL) using various pre-trained commercial and open-source models. Our findings reveal that ICL achieves the best performance, with commercial models like GPT-3.5 and GPT-4 attaining the best performance. However, there remains a significant 14 percentage points gap between our best few-shot learning score and the performance of HerBERT-large fine-tuned on the entire training dataset. Among the techniques, SetFit emerges as the second-best approach, closely followed by linear probing. We observed the worst and most unstable performance with non-linear head fine-tuning. Results for ICL indicate that continual pre-training of models like Mistral-7b or Llama-2-13b on Polish corpora is beneficial. This is confirmed by the improved performances of Bielik-7b and Trurl-13b, respectively. To further support experiments in few-shot learning for Polish, we are releasing handcrafted templates for the ICL.<|reference_end|>
@article{hadeliya2024evaluation,
    title={Evaluation of Few-Shot Learning for Classification Tasks in the Polish
  Language},
    author={Tsimur Hadeliya, Dariusz Kajtoch},
    journal={arXiv preprint arXiv:2404.17832},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.17832},
    primaryClass={cs.CL}
}