title: Copy Is All You Need
summary: The dominant text generation models compose the output by sequentially
  selecting words from a fixed vocabulary. In this paper, we formulate text
  generation as progressively copying text segments (e.g., words or phrases) from
  an existing text collection. We compute the contextualized representations of
  meaningful text segments and index them using efficient vector search toolkits.
  The task of text generation is then decomposed into a series of copy-and-paste
  operations: at each time step, we seek suitable text spans from the text
  collection rather than selecting from a standalone vocabulary. Experiments on
  the standard language modeling benchmark (WikiText-103) show that our approach
  achieves better generation quality according to both automatic and human
  evaluations. Besides, its inference efficiency is comparable to token-level
  autoregressive models thanks to the reduction of decoding steps. We also show
  that our approach allows for effective domain adaptation by simply switching to
  domain-specific text collection without extra training. Finally, we observe
  that our approach attains additional performance gains by simply scaling up to
  larger text collections, again without further training.Our source
  codes are publicly available at
  \url{https://github.com/gmftbyGMFTBY/Copyisallyouneed.}
paper_authors._id: 64b48f5993679cd0095d8b5a
paper_authors.name: Tian Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-25T08:08:18.837Z
paper_authors._id: 627ca439a09edfe62239c671
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652335660508-noauth.jpeg
paper_authors.fullname: Tian Lan
paper_authors.type: user
paper_authors.user: GMFTBY
paper_authors._id: 64b48f5993679cd0095d8b5b
paper_authors.name: Deng Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T10:35:39.765Z
paper_authors._id: 646735a1946476c5d214e5b2
paper_authors.avatarUrl: /avatars/23392fda23fa6288fcb309f7eaa73472.svg
paper_authors.fullname: Deng Cai
paper_authors.type: user
paper_authors.user: jcyk
paper_authors._id: 64b48f5993679cd0095d8b5c
paper_authors.name: Yan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-24T07:44:19.066Z
paper_authors._id: 64674375946476c5d215382d
paper_authors.avatarUrl: /avatars/0f0b98080c064a17d003da8a22a047cc.svg
paper_authors.fullname: Yan Wang
paper_authors.type: user
paper_authors.user: libertywing
paper_authors._id: 64b48f5993679cd0095d8b5d
paper_authors.name: Heyan Huang
paper_authors._id: 64b48f5993679cd0095d8b5e
paper_authors.name: Xian-Ling Mao
paper_title: Copy Is All You Need
paper_summary: The dominant text generation models compose the output by sequentially
  selecting words from a fixed vocabulary. In this paper, we formulate text
  generation as progressively copying text segments (e.g., words or phrases) from
  an existing text collection. We compute the contextualized representations of
  meaningful text segments and index them using efficient vector search toolkits.
  The task of text generation is then decomposed into a series of copy-and-paste
  operations: at each time step, we seek suitable text spans from the text
  collection rather than selecting from a standalone vocabulary. Experiments on
  the standard language modeling benchmark (WikiText-103) show that our approach
  achieves better generation quality according to both automatic and human
  evaluations. Besides, its inference efficiency is comparable to token-level
  autoregressive models thanks to the reduction of decoding steps. We also show
  that our approach allows for effective domain adaptation by simply switching to
  domain-specific text collection without extra training. Finally, we observe
  that our approach attains additional performance gains by simply scaling up to
  larger text collections, again without further training.Our source
  codes are publicly available at
  \url{https://github.com/gmftbyGMFTBY/Copyisallyouneed.}
paper_ai_keywords: contextualized representations vector search toolkits copy-and-paste operations text spans language modeling benchmark (WikiText-103) automatic evaluations human evaluations inference efficiency token-level autoregressive models domain adaptation domain-specific text collection performance gains scaling up