Rethinking the adaptive relationship between Encoder Layers and Decoder Layers
<|reference_start|>Rethinking the adaptive relationship between Encoder Layers and Decoder Layers: This article explores the adaptive relationship between Encoder Layers and Decoder Layers using the SOTA model Helsinki-NLP/opus-mt-de-en, which translates German to English. The specific method involves introducing a bias-free fully connected layer between the Encoder and Decoder, with different initializations of the layer's weights, and observing the outcomes of fine-tuning versus retraining. Four experiments were conducted in total. The results suggest that directly modifying the pre-trained model structure for fine-tuning yields suboptimal performance. However, upon observing the outcomes of the experiments with retraining, this structural adjustment shows significant potential.<|reference_end|>
@article{song2024rethinking,
    title={Rethinking the adaptive relationship between Encoder Layers and Decoder
  Layers},
    author={Yubo Song},
    journal={arXiv preprint arXiv:2405.08570},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.08570},
    primaryClass={cs.CL}
}