On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems
<|reference_start|>On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for Few-shot Problems: Large Language Models (LLMs) have been observed to perform well on a wide range of downstream tasks when fine-tuned on domain-specific data. However, such data may not be readily available in many applications, motivating zero-shot or few-shot approaches using domain-adjacent models. While several fine-tuned models for various tasks are available, finding an appropriate domain-adjacent model for a given task is often not straight forward. In this paper, we study DAFT-E, a framework that utilizes an Ensemble of Domain-Adjacent Fine-Tuned Foundation Models for few-shot problems. We show that for zero-shot problems, this ensembling method provides an accuracy performance close to that of the single best model. With few-shot problems, this performance improves further, at which point DEFT-E can outperform any single domain-adjacent model while requiring much less data for domain-specific fine-tuning.<|reference_end|>
@article{alam2024on,
    title={On the Utility of Domain-Adjacent Fine-Tuned Model Ensembles for
  Few-shot Problems},
    author={Md Ibrahim Ibne Alam, Parikshit Ram, Soham Dan, Horst Samulowitz,
  Koushik Kar},
    journal={arXiv preprint arXiv:2406.13720},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.13720},
    primaryClass={cs.CL cs.LG}
}