title: Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
summary: Recent studies have shown that dual encoder models trained with the
  sentence-level translation ranking task are effective methods for cross-lingual
  sentence embedding. However, our research indicates that token-level alignment
  is also crucial in multilingual scenarios, which has not been fully explored
  previously. Based on our findings, we propose a dual-alignment pre-training
  (DAP) framework for cross-lingual sentence embedding that incorporates both
  sentence-level and token-level alignment. To achieve this, we introduce a novel
  representation translation learning (RTL) task, where the model learns to use
  one-side contextualized token representation to reconstruct its translation
  counterpart. This reconstruction objective encourages the model to embed
  translation information into the token representation. Compared to other
  token-level alignment methods such as translation language modeling, RTL is
  more suitable for dual encoder architectures and is computationally efficient.
  Extensive experiments on three sentence-level cross-lingual benchmarks
  demonstrate that our approach can significantly improve sentence embedding. Our
  code is available at https://github.com/ChillingDream/DAP.
paper_authors._id: 646428a8e0475a83333db26a
paper_authors.name: Ziheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T09:35:16.658Z
paper_authors._id: 636a7459eb076ec3f4030e7d
paper_authors.avatarUrl: /avatars/832dc709211e3a2ea5e93caea3768122.svg
paper_authors.fullname: Ziheng Li
paper_authors.type: user
paper_authors.user: ChillingDream
paper_authors._id: 646428a8e0475a83333db26b
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:25:41.595Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 646428a8e0475a83333db26c
paper_authors.name: Zihan Zhang
paper_authors._id: 646428a8e0475a83333db26d
paper_authors.name: Zhi-Hong Deng
paper_authors._id: 646428a8e0475a83333db26e
paper_authors.name: Qiang Lou
paper_authors._id: 646428a8e0475a83333db26f
paper_authors.name: Haizhen Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:27:11.562Z
paper_authors._id: 6302e8e57e137e3d6b3f5a06
paper_authors.avatarUrl: /avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Haizhen
paper_authors._id: 646428a8e0475a83333db270
paper_authors.name: Jian Jiao
paper_authors._id: 646428a8e0475a83333db271
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:27:40.995Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 646428a8e0475a83333db272
paper_authors.name: Weiwei Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:27:50.802Z
paper_authors._id: 642da49dc1adada5b345c72b
paper_authors.avatarUrl: /avatars/5fa8d345a7248246454444999756105b.svg
paper_authors.fullname: deng
paper_authors.type: user
paper_authors.user: weiweideng
paper_authors._id: 646428a8e0475a83333db273
paper_authors.name: Qi Zhang
paper_title: Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
paper_summary: Recent studies have shown that dual encoder models trained with the
  sentence-level translation ranking task are effective methods for cross-lingual
  sentence embedding. However, our research indicates that token-level alignment
  is also crucial in multilingual scenarios, which has not been fully explored
  previously. Based on our findings, we propose a dual-alignment pre-training
  (DAP) framework for cross-lingual sentence embedding that incorporates both
  sentence-level and token-level alignment. To achieve this, we introduce a novel
  representation translation learning (RTL) task, where the model learns to use
  one-side contextualized token representation to reconstruct its translation
  counterpart. This reconstruction objective encourages the model to embed
  translation information into the token representation. Compared to other
  token-level alignment methods such as translation language modeling, RTL is
  more suitable for dual encoder architectures and is computationally efficient.
  Extensive experiments on three sentence-level cross-lingual benchmarks
  demonstrate that our approach can significantly improve sentence embedding. Our
  code is available at https://github.com/ChillingDream/DAP.
paper_ai_keywords: dual encoder models sentence-level translation ranking task token-level alignment cross-lingual sentence embedding dual-alignment pre-training (DAP) framework representation translation learning (RTL) task one-side contextualized token representation translation counterpart translation information translation language modeling computational efficiency sentence-level cross-lingual benchmarks