title: mGTE: Generalized Long-Context Text Representation and Reranking Models
    for Multilingual Text Retrieval
summary: We present systematic efforts in building long-context multilingual text
  representation model (TRM) and reranker from scratch for text retrieval. We
  first introduce a text encoder (base size) enhanced with RoPE and unpadding,
  pre-trained in a native 8192-token context (longer than 512 of previous
  multilingual encoders). Then we construct a hybrid TRM and a cross-encoder
  reranker by contrastive learning. Evaluations show that our text encoder
  outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM
  and reranker match the performance of large-sized state-of-the-art BGE-M3
  models and achieve better results on long-context retrieval benchmarks. Further
  analysis demonstrate that our proposed models exhibit higher efficiency during
  both training and inference. We believe their efficiency and effectiveness
  could benefit various researches and industrial applications.
paper_authors._id: 66a83d59bb22d7e78a42ec7b
paper_authors.name: Xin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T07:25:59.454Z
paper_authors._id: 63b6dbc8ccebeadccc888456
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673396893898-63b6dbc8ccebeadccc888456.jpeg
paper_authors.fullname: Xin Zhang
paper_authors.type: user
paper_authors.user: izhx
paper_authors._id: 66a83d59bb22d7e78a42ec7c
paper_authors.name: Yanzhao Zhang
paper_authors._id: 66a83d59bb22d7e78a42ec7d
paper_authors.name: Dingkun Long
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T07:25:57.779Z
paper_authors._id: 616adb8578833ce5997e441a
paper_authors.avatarUrl: /avatars/bf5c04a6032709f35e3fb48e1be6976f.svg
paper_authors.fullname: Dingkun Long
paper_authors.type: user
paper_authors.user: thenlper
paper_authors._id: 66a83d59bb22d7e78a42ec7e
paper_authors.name: Wen Xie
paper_authors._id: 66a83d59bb22d7e78a42ec7f
paper_authors.name: Ziqi Dai
paper_authors._id: 66a83d59bb22d7e78a42ec80
paper_authors.name: Jialong Tang
paper_authors._id: 66a83d59bb22d7e78a42ec81
paper_authors.name: Huan Lin
paper_authors._id: 66a83d59bb22d7e78a42ec82
paper_authors.name: Baosong Yang
paper_authors._id: 66a83d59bb22d7e78a42ec83
paper_authors.name: Pengjun Xie
paper_authors._id: 66a83d59bb22d7e78a42ec84
paper_authors.name: Fei Huang
paper_authors._id: 66a83d59bb22d7e78a42ec85
paper_authors.name: Meishan Zhang
paper_authors._id: 66a83d59bb22d7e78a42ec86
paper_authors.name: Wenjie Li
paper_authors._id: 66a83d59bb22d7e78a42ec87
paper_authors.name: Min Zhang
paper_title: mGTE: Generalized Long-Context Text Representation and Reranking Models
    for Multilingual Text Retrieval
paper_summary: We present systematic efforts in building long-context multilingual text
  representation model (TRM) and reranker from scratch for text retrieval. We
  first introduce a text encoder (base size) enhanced with RoPE and unpadding,
  pre-trained in a native 8192-token context (longer than 512 of previous
  multilingual encoders). Then we construct a hybrid TRM and a cross-encoder
  reranker by contrastive learning. Evaluations show that our text encoder
  outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM
  and reranker match the performance of large-sized state-of-the-art BGE-M3
  models and achieve better results on long-context retrieval benchmarks. Further
  analysis demonstrate that our proposed models exhibit higher efficiency during
  both training and inference. We believe their efficiency and effectiveness
  could benefit various researches and industrial applications.
paper_ai_keywords: long-context multilingual text representation model (TRM) reranker text encoder RoPE unpadding native 8192-token context hybrid TRM cross-encoder contrastive learning XLM-R BGE-M3 long-context retrieval benchmarks