ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG
  Capabilities
In this work, we introduce ChatQA 2, a Llama3-based model designed to bridge
the gap between open-access LLMs and leading proprietary models (e.g.,
GPT-4-Turbo) in long-context understanding and retrieval-augmented generation
(RAG) capabilities. These two capabilities are essential for LLMs to process
large volumes of information that cannot fit into a single prompt and are
complementary to each other, depending on the downstream tasks and
computational budgets. We present a detailed continued training recipe to
extend the context window of Llama3-70B-base from 8K to 128K tokens, along with
a three-stage instruction tuning process to enhance the model's
instruction-following, RAG performance, and long-context understanding
capabilities. Our results demonstrate that the Llama3-ChatQA-2-70B model
achieves accuracy comparable to GPT-4-Turbo-2024-0409 on many long-context
understanding tasks and surpasses it on the RAG benchmark. Interestingly, we
find that the state-of-the-art long-context retriever can alleviate the top-k
context fragmentation issue in RAG, further improving RAG-based results for
long-context understanding tasks. We also provide extensive comparisons between
RAG and long-context solutions using state-of-the-art long-context LLMs.
669dcf02c517d804cf9ad7b2
Peng Xu
669dcf02c517d804cf9ad7b3
Wei Ping
669dcf02c517d804cf9ad7b4
Xianchao Wu
admin_assigned
2024-07-22T07:29:45.409Z
62075297bcffa036eb483b2b
https://cdn-avatars.huggingface.co/v1/production/uploads/1644647204919-62075297bcffa036eb483b2b.jpeg
Xianchao Wu
user
wuxianchao
669dcf02c517d804cf9ad7b5
Zihan Liu
admin_assigned
2024-07-22T07:30:15.794Z
65f33b1c9f7970ccc0234cbf
/avatars/99fbab303912e3674663251c04279907.svg
Zihan Liu
user
zihanliu
669dcf02c517d804cf9ad7b6
Mohammad Shoeybi
admin_assigned
2024-07-22T07:30:22.073Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
669dcf02c517d804cf9ad7b7
Bryan Catanzaro
admin_assigned
2024-07-22T07:30:29.898Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG
  Capabilities
In this work, we introduce ChatQA 2, a Llama3-based model designed to bridge
the gap between open-access LLMs and leading proprietary models (e.g.,
GPT-4-Turbo) in long-context understanding and retrieval-augmented generation
(RAG) capabilities. These two capabilities are essential for LLMs to process
large volumes of information that cannot fit into a single prompt and are
complementary to each other, depending on the downstream tasks and
computational budgets. We present a detailed continued training recipe to
extend the context window of Llama3-70B-base from 8K to 128K tokens, along with
a three-stage instruction tuning process to enhance the model's
instruction-following, RAG performance, and long-context understanding
capabilities. Our results demonstrate that the Llama3-ChatQA-2-70B model
achieves accuracy comparable to GPT-4-Turbo-2024-0409 on many long-context
understanding tasks and surpasses it on the RAG benchmark. Interestingly, we
find that the state-of-the-art long-context retriever can alleviate the top-k
context fragmentation issue in RAG, further improving RAG-based results for
long-context understanding tasks. We also provide extensive comparisons between
RAG and long-context solutions using state-of-the-art long-context LLMs.
ChatQA 2, a Llama3-based model, extends the context window to 128K tokens and enhances RAG performance, achieving accuracy comparable to GPT-4-Turbo on long-context tasks and surpassing it on RAG benchmarks.
Llama3 long-context understanding retrieval-augmented generation RAG continued training instruction tuning context window top-k context fragmentation long-context retriever