title: Selecting Influential Samples for Long Context Alignment via Homologous
    Models' Guidance and Contextual Awareness Measurement
summary: The expansion of large language models to effectively handle instructions
  with extremely long contexts has yet to be fully investigated. The primary
  obstacle lies in constructing a high-quality long instruction-following dataset
  devised for long context alignment. Existing studies have attempted to scale up
  the available data volume by synthesizing long instruction-following samples.
  However, indiscriminately increasing the quantity of data without a
  well-defined strategy for ensuring data quality may introduce low-quality
  samples and restrict the final performance. To bridge this gap, we aim to
  address the unique challenge of long-context alignment, i.e., modeling the
  long-range dependencies for handling instructions and lengthy input contexts.
  We propose GATEAU, a novel framework designed to identify the influential and
  high-quality samples enriched with long-range dependency relations by utilizing
  crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement
  (CAM). Specifically, HMG attempts to measure the difficulty of generating
  corresponding responses due to the long-range dependencies, using the
  perplexity scores of the response from two homologous models with different
  context windows. Also, the role of CAM is to measure the difficulty of
  understanding the long input contexts due to long-range dependencies by
  evaluating whether the model's attention is focused on important segments.
  Built upon both proposed methods, we select the most challenging samples as the
  influential data to effectively frame the long-range dependencies, thereby
  achieving better performance of LLMs. Comprehensive experiments indicate that
  GATEAU effectively identifies samples enriched with long-range dependency
  relations and the model trained on these selected samples exhibits better
  instruction-following and long-context understanding capabilities.
paper_authors._id: 671742ab0c095b8e5d099551
paper_authors.name: Shuzheng Si
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T15:02:35.806Z
paper_authors._id: 637c99bbfe115289cfedfb44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
paper_authors.fullname: ssz
paper_authors.type: user
paper_authors.user: ssz1111
paper_authors._id: 671742ab0c095b8e5d099552
paper_authors.name: Haozhe Zhao
paper_authors._id: 671742ab0c095b8e5d099553
paper_authors.name: Gang Chen
paper_authors._id: 671742ab0c095b8e5d099554
paper_authors.name: Yunshui Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:29:52.787Z
paper_authors._id: 62e670d33651180f7d334ef3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661829952739-62e670d33651180f7d334ef3.jpeg
paper_authors.fullname: LiYunshui
paper_authors.type: user
paper_authors.user: Wa2erGo
paper_authors._id: 671742ab0c095b8e5d099555
paper_authors.name: Kangyang Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:29:58.585Z
paper_authors._id: 66ba08d4bafd993a5428e051
paper_authors.avatarUrl: /avatars/2bd7779582ee3ae2ff9688e1627d745e.svg
paper_authors.fullname: Kangyang Luo
paper_authors.type: user
paper_authors.user: lKangyang
paper_authors._id: 671742ab0c095b8e5d099556
paper_authors.name: Chuancheng Lv
paper_authors._id: 671742ab0c095b8e5d099557
paper_authors.name: Kaikai An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T07:34:44.942Z
paper_authors._id: 66c89152d33e34fbc29497d7
paper_authors.avatarUrl: /avatars/bbddabf6532393951c4759e5915a065b.svg
paper_authors.fullname: KaikaiAn
paper_authors.type: user
paper_authors.user: kkk-an
paper_authors._id: 671742ab0c095b8e5d099558
paper_authors.name: Fanchao Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T13:29:21.938Z
paper_authors._id: 65623af21a0c459616cc4b03
paper_authors.avatarUrl: /avatars/76467e8a3c46aca3687ffcfe964a2889.svg
paper_authors.fullname: Fanchao Qi
paper_authors.type: user
paper_authors.user: fanchao-qi
paper_authors._id: 671742ab0c095b8e5d099559
paper_authors.name: Baobao Chang
paper_authors._id: 671742ab0c095b8e5d09955a
paper_authors.name: Maosong Sun
paper_title: Selecting Influential Samples for Long Context Alignment via Homologous
    Models' Guidance and Contextual Awareness Measurement
paper_summary: The expansion of large language models to effectively handle instructions
  with extremely long contexts has yet to be fully investigated. The primary
  obstacle lies in constructing a high-quality long instruction-following dataset
  devised for long context alignment. Existing studies have attempted to scale up
  the available data volume by synthesizing long instruction-following samples.
  However, indiscriminately increasing the quantity of data without a
  well-defined strategy for ensuring data quality may introduce low-quality
  samples and restrict the final performance. To bridge this gap, we aim to
  address the unique challenge of long-context alignment, i.e., modeling the
  long-range dependencies for handling instructions and lengthy input contexts.
  We propose GATEAU, a novel framework designed to identify the influential and
  high-quality samples enriched with long-range dependency relations by utilizing
  crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement
  (CAM). Specifically, HMG attempts to measure the difficulty of generating
  corresponding responses due to the long-range dependencies, using the
  perplexity scores of the response from two homologous models with different
  context windows. Also, the role of CAM is to measure the difficulty of
  understanding the long input contexts due to long-range dependencies by
  evaluating whether the model's attention is focused on important segments.
  Built upon both proposed methods, we select the most challenging samples as the
  influential data to effectively frame the long-range dependencies, thereby
  achieving better performance of LLMs. Comprehensive experiments indicate that
  GATEAU effectively identifies samples enriched with long-range dependency
  relations and the model trained on these selected samples exhibits better
  instruction-following and long-context understanding capabilities.
paper_ai_keywords: long language models long contexts instruction-following dataset long context alignment Homologous Models' Guidance (HMG) Contextual Awareness Measurement (CAM) perplexity scores long-range dependencies attention instruction-following and long-context understanding capabilities