Bitune: Bidirectional Instruction-Tuning
<|reference_start|>Bitune: Bidirectional Instruction-Tuning: We introduce Bitune, a method that improves instruction-tuning of pretrained decoder-only large language models, leading to consistent gains on downstream tasks. Bitune applies both causal and bidirectional attention to the prompt, to obtain a better representation of the query or instruction. We realize this by introducing two sets of parameters, for which we apply parameter-efficient finetuning techniques. These causal and bidirectional features are then combined into a weighted average with trainable coefficients, which is subsequently used to generate new tokens. We demonstrate significant improvements in zero-shot performance on commonsense reasoning, arithmetic, and language understanding tasks, while extensive ablation studies validate the role of each component and demonstrate the method's agnosticism to different PEFT techniques.<|reference_end|>
@article{kopiczko2024bitune:,
    title={Bitune: Bidirectional Instruction-Tuning},
    author={Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano},
    journal={arXiv preprint arXiv:2405.14862},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14862},
    primaryClass={cs.CL}
}