mGTE: Generalized Long-Context Text Representation and Reranking Models
  for Multilingual Text Retrieval
We present systematic efforts in building long-context multilingual text
representation model (TRM) and reranker from scratch for text retrieval. We
first introduce a text encoder (base size) enhanced with RoPE and unpadding,
pre-trained in a native 8192-token context (longer than 512 of previous
multilingual encoders). Then we construct a hybrid TRM and a cross-encoder
reranker by contrastive learning. Evaluations show that our text encoder
outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM
and reranker match the performance of large-sized state-of-the-art BGE-M3
models and achieve better results on long-context retrieval benchmarks. Further
analysis demonstrate that our proposed models exhibit higher efficiency during
both training and inference. We believe their efficiency and effectiveness
could benefit various researches and industrial applications.
66a83d59bb22d7e78a42ec7b
Xin Zhang
claimed_verified
2024-07-30T07:25:59.454Z
63b6dbc8ccebeadccc888456
https://cdn-avatars.huggingface.co/v1/production/uploads/1673396893898-63b6dbc8ccebeadccc888456.jpeg
Xin Zhang
user
izhx
66a83d59bb22d7e78a42ec7c
Yanzhao Zhang
66a83d59bb22d7e78a42ec7d
Dingkun Long
claimed_verified
2024-07-30T07:25:57.779Z
616adb8578833ce5997e441a
/avatars/bf5c04a6032709f35e3fb48e1be6976f.svg
Dingkun Long
user
thenlper
66a83d59bb22d7e78a42ec7e
Wen Xie
66a83d59bb22d7e78a42ec7f
Ziqi Dai
66a83d59bb22d7e78a42ec80
Jialong Tang
66a83d59bb22d7e78a42ec81
Huan Lin
66a83d59bb22d7e78a42ec82
Baosong Yang
66a83d59bb22d7e78a42ec83
Pengjun Xie
66a83d59bb22d7e78a42ec84
Fei Huang
66a83d59bb22d7e78a42ec85
Meishan Zhang
66a83d59bb22d7e78a42ec86
Wenjie Li
66a83d59bb22d7e78a42ec87
Min Zhang
mGTE: Generalized Long-Context Text Representation and Reranking Models
  for Multilingual Text Retrieval
We present systematic efforts in building long-context multilingual text
representation model (TRM) and reranker from scratch for text retrieval. We
first introduce a text encoder (base size) enhanced with RoPE and unpadding,
pre-trained in a native 8192-token context (longer than 512 of previous
multilingual encoders). Then we construct a hybrid TRM and a cross-encoder
reranker by contrastive learning. Evaluations show that our text encoder
outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM
and reranker match the performance of large-sized state-of-the-art BGE-M3
models and achieve better results on long-context retrieval benchmarks. Further
analysis demonstrate that our proposed models exhibit higher efficiency during
both training and inference. We believe their efficiency and effectiveness
could benefit various researches and industrial applications.
A long-context multilingual text representation model and reranker are developed and evaluated, showing superior performance and efficiency compared to existing models.
RoPE unpadding contrastive learning hybrid TRM cross-encoder reranker XLM-R BGE-M3 long-context retrieval