title: Masking in Multi-hop QA: An Analysis of How Language Models Perform with
    Context Permutation
summary: Multi-hop Question Answering (MHQA) adds layers of complexity to question
  answering, making it more challenging. When Language Models (LMs) are prompted
  with multiple search results, they are tasked not only with retrieving relevant
  information but also employing multi-hop reasoning across the information
  sources. Although LMs perform well on traditional question-answering tasks, the
  causal mask can hinder their capacity to reason across complex contexts. In
  this paper, we explore how LMs respond to multi-hop questions by permuting
  search results (retrieved documents) under various configurations. Our study
  reveals interesting findings as follows: 1) Encoder-decoder models, such as the
  ones in the Flan-T5 family, generally outperform causal decoder-only LMs in
  MHQA tasks, despite being significantly smaller in size; 2) altering the order
  of gold documents reveals distinct trends in both Flan T5 models and fine-tuned
  decoder-only models, with optimal performance observed when the document order
  aligns with the reasoning chain order; 3) enhancing causal decoder-only models
  with bi-directional attention by modifying the causal mask can effectively
  boost their end performance. In addition to the above, we conduct a thorough
  investigation of the distribution of LM attention weights in the context of
  MHQA. Our experiments reveal that attention weights tend to peak at higher
  values when the resulting answer is correct. We leverage this finding to
  heuristically improve LMs' performance on this task. Our code is publicly
  available at https://github.com/hwy9855/MultiHopQA-Reasoning.
paper_authors._id: 682bdc203fff9e115285b2b6
paper_authors.name: Wenyu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:21:08.965Z
paper_authors._id: 64c4d03ca684146b1cdb8237
paper_authors.avatarUrl: /avatars/82b513c686ab20c859ca900fd59dfac7.svg
paper_authors.fullname: Wenyu Huang
paper_authors.type: user
paper_authors.user: hwy9855
paper_authors._id: 682bdc203fff9e115285b2b7
paper_authors.name: Pavlos Vougiouklis
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-21T11:43:19.081Z
paper_authors._id: 641c83a492cd2530299714c7
paper_authors.avatarUrl: /avatars/928195163f9040a2649953c86b9d6c79.svg
paper_authors.fullname: Pavlos V
paper_authors.type: user
paper_authors.user: pvougiou
paper_authors._id: 682bdc203fff9e115285b2b8
paper_authors.name: Mirella Lapata
paper_authors._id: 682bdc203fff9e115285b2b9
paper_authors.name: Jeff Z. Pan
paper_title: Masking in Multi-hop QA: An Analysis of How Language Models Perform with
    Context Permutation
paper_summary: Multi-hop Question Answering (MHQA) adds layers of complexity to question
  answering, making it more challenging. When Language Models (LMs) are prompted
  with multiple search results, they are tasked not only with retrieving relevant
  information but also employing multi-hop reasoning across the information
  sources. Although LMs perform well on traditional question-answering tasks, the
  causal mask can hinder their capacity to reason across complex contexts. In
  this paper, we explore how LMs respond to multi-hop questions by permuting
  search results (retrieved documents) under various configurations. Our study
  reveals interesting findings as follows: 1) Encoder-decoder models, such as the
  ones in the Flan-T5 family, generally outperform causal decoder-only LMs in
  MHQA tasks, despite being significantly smaller in size; 2) altering the order
  of gold documents reveals distinct trends in both Flan T5 models and fine-tuned
  decoder-only models, with optimal performance observed when the document order
  aligns with the reasoning chain order; 3) enhancing causal decoder-only models
  with bi-directional attention by modifying the causal mask can effectively
  boost their end performance. In addition to the above, we conduct a thorough
  investigation of the distribution of LM attention weights in the context of
  MHQA. Our experiments reveal that attention weights tend to peak at higher
  values when the resulting answer is correct. We leverage this finding to
  heuristically improve LMs' performance on this task. Our code is publicly
  available at https://github.com/hwy9855/MultiHopQA-Reasoning.
paper_ai_summary: Encoder-decoder models outperform causal decoder-only models in multi-hop question answering by leveraging permutation of search results and bi-directional attention.
paper_ai_keywords: multi-hop question answering encoder-decoder models causal decoder-only models Flan-T5 multi-hop reasoning causal mask bi-directional attention attention weights