HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labelling
<|reference_start|>HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labelling: We present HyperLoader, a simple approach that combines different parameter-efficient fine-tuning methods in a multi-task setting. To achieve this goal, our model uses a hypernetwork to generate the weights of these modules based on the task, the transformer layer, and its position within this layer. Our method combines the benefits of multi-task learning by capturing the structure of all tasks while reducing the task interference problem by encapsulating the task-specific knowledge in the generated weights and the benefits of combining different parameter-efficient methods to outperform full-fine tuning. We provide empirical evidence that HyperLoader outperforms previous approaches in most datasets and obtains the best average performance across tasks in high-resource and low-resource scenarios.<|reference_end|>
@article{ortiz-barajas2024hyperloader:,
    title={HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into
  Multi-Task Transformers for Sequence Labelling},
    author={Jesus-German Ortiz-Barajas, Helena Gomez-Adorno, Thamar Solorio},
    journal={arXiv preprint arXiv:2407.01411},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01411},
    primaryClass={cs.CL}
}