Fisher Mask Nodes for Language Model Merging
<|reference_start|>Fisher Mask Nodes for Language Model Merging: Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growing field of model merging provides a solution, dealing with the challenge of combining multiple task-specific models into a single multi-task model. In this study, we introduce a novel model merging method for Transformers, combining insights from previous work in Fisher-weighted averaging and the use of Fisher information in model pruning. Utilizing the Fisher information of mask nodes within the Transformer architecture, we devise a computationally efficient weighted-averaging scheme. Our method exhibits a regular and significant performance increase across various models in the BERT family, outperforming full-scale Fisher-weighted averaging in a fraction of the computational cost, with baseline performance improvements of up to +6.5 and a speedup between 57.4x and 321.7x across models. Our results prove the potential of our method in current multi-task learning environments and suggest its scalability and adaptability to new model architectures and learning scenarios.<|reference_end|>
@article{k2024fisher,
    title={Fisher Mask Nodes for Language Model Merging},
    author={Thennal D K, Ganesh Nathan, Suchithra M S},
    journal={arXiv preprint arXiv:2403.09891},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.09891},
    primaryClass={cs.CL cs.AI cs.LG}
}