Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond
<|reference_start|>Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond: Task embedding, a meta-learning technique that captures task-specific information, has gained popularity, especially in areas such as multi-task learning, model editing, and interpretability. However, it faces challenges with the emergence of prompt-guided Large Language Models (LLMs) operating in a gradient-free manner. Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders the adaptability of task embeddings across diverse models, especially prompt-based LLMs. To hardness the potential of task embeddings in the era of LLMs, we propose a framework for unified task embeddings (FUTE), harmonizing task embeddings from various models, including smaller language models and LLMs with varied prompts, within a single vector space. Such uniformity enables comparison and analysis of similarities amongst different models, broadening the scope and utility of existing task embedding methods in multi-model scenarios, while maintaining their performance comparable to architecture-specific methods.<|reference_end|>
@article{wang2024towards,
    title={Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap
  for Prompt-Based Large Language Models and Beyond},
    author={Xinyu Wang, Hainiu Xu, Lin Gui, Yulan He},
    journal={arXiv preprint arXiv:2402.14522},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.14522},
    primaryClass={cs.CL cs.LG}
}