Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Recent studies have shown that dual encoder models trained with the
sentence-level translation ranking task are effective methods for cross-lingual
sentence embedding. However, our research indicates that token-level alignment
is also crucial in multilingual scenarios, which has not been fully explored
previously. Based on our findings, we propose a dual-alignment pre-training
(DAP) framework for cross-lingual sentence embedding that incorporates both
sentence-level and token-level alignment. To achieve this, we introduce a novel
representation translation learning (RTL) task, where the model learns to use
one-side contextualized token representation to reconstruct its translation
counterpart. This reconstruction objective encourages the model to embed
translation information into the token representation. Compared to other
token-level alignment methods such as translation language modeling, RTL is
more suitable for dual encoder architectures and is computationally efficient.
Extensive experiments on three sentence-level cross-lingual benchmarks
demonstrate that our approach can significantly improve sentence embedding. Our
code is available at https://github.com/ChillingDream/DAP.
646428a8e0475a83333db26a
Ziheng Li
claimed_verified
2023-05-17T09:35:16.658Z
636a7459eb076ec3f4030e7d
/avatars/832dc709211e3a2ea5e93caea3768122.svg
Ziheng Li
user
ChillingDream
646428a8e0475a83333db26b
Shaohan Huang
admin_assigned
2023-05-26T09:25:41.595Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
646428a8e0475a83333db26c
Zihan Zhang
646428a8e0475a83333db26d
Zhi-Hong Deng
646428a8e0475a83333db26e
Qiang Lou
646428a8e0475a83333db26f
Haizhen Huang
admin_assigned
2023-05-26T09:27:11.562Z
6302e8e57e137e3d6b3f5a06
/avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
Huang
user
Haizhen
646428a8e0475a83333db270
Jian Jiao
646428a8e0475a83333db271
Furu Wei
admin_assigned
2023-05-26T09:27:40.995Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
646428a8e0475a83333db272
Weiwei Deng
admin_assigned
2023-05-26T09:27:50.802Z
642da49dc1adada5b345c72b
/avatars/5fa8d345a7248246454444999756105b.svg
deng
user
weiweideng
646428a8e0475a83333db273
Qi Zhang
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Recent studies have shown that dual encoder models trained with the
sentence-level translation ranking task are effective methods for cross-lingual
sentence embedding. However, our research indicates that token-level alignment
is also crucial in multilingual scenarios, which has not been fully explored
previously. Based on our findings, we propose a dual-alignment pre-training
(DAP) framework for cross-lingual sentence embedding that incorporates both
sentence-level and token-level alignment. To achieve this, we introduce a novel
representation translation learning (RTL) task, where the model learns to use
one-side contextualized token representation to reconstruct its translation
counterpart. This reconstruction objective encourages the model to embed
translation information into the token representation. Compared to other
token-level alignment methods such as translation language modeling, RTL is
more suitable for dual encoder architectures and is computationally efficient.
Extensive experiments on three sentence-level cross-lingual benchmarks
demonstrate that our approach can significantly improve sentence embedding. Our
code is available at https://github.com/ChillingDream/DAP.
A dual-alignment pre-training framework improves cross-lingual sentence embedding by integrating both sentence-level and token-level alignment using representation translation learning.
dual encoder models sentence-level translation ranking task token-level alignment dual-alignment pre-training (DAP) representation translation learning (RTL) translation language modeling cross-lingual benchmarks