  Document structure is critical for efficient information consumption.
However, it is challenging to encode it efficiently into the modern Transformer
architecture. In this work, we present HIBRIDS, which injects Hierarchical
Biases foR Incorporating Document Structure into the calculation of attention
scores. We further present a new task, hierarchical question-summary
generation, for summarizing salient content in the source document into a
hierarchy of questions and summaries, where each follow-up question inquires
about the content of its parent question-summary pair. We also annotate a new
dataset with 6,153 question-summary hierarchies labeled on long government
reports. Experiment results show that our model produces better
question-summary hierarchies than comparisons on both hierarchy quality and
content coverage, a finding also echoed by human judges. Additionally, our
model improves the generation of long-form summaries from lengthy government
reports and Wikipedia articles, as measured by ROUGE scores.

Computation and Language