title: SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced
    Token Detection
summary: Pre-training large language models is known to be extremely resource
  intensive and often times inefficient, under-utilizing the information
  encapsulated in the training text sequences. In this paper, we present SpacTor,
  a new training procedure consisting of (1) a hybrid objective combining span
  corruption (SC) and token replacement detection (RTD), and (2) a two-stage
  curriculum that optimizes the hybrid objective over the initial tau
  iterations, then transitions to standard SC loss. We show empirically that the
  effectiveness of the hybrid objective is tied to the two-stage pre-training
  schedule, and provide extensive analysis on why this is the case. In our
  experiments with encoder-decoder architectures (T5) on a variety of NLP tasks,
  SpacTor-T5 yields the same downstream performance as standard SC pre-training,
  while enabling a 50% reduction in pre-training iterations and 40% reduction in
  total FLOPs. Alternatively, given the same amount of computing budget, we find
  that SpacTor results in significantly improved downstream benchmark
  performance.
paper_authors._id: 65b1e70254fa188a2fbf4852
paper_authors.name: Ke Ye
paper_authors._id: 65b1e70254fa188a2fbf4853
paper_authors.name: Heinrich Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:23:19.387Z
paper_authors._id: 64628caf77f63463fc5f8d6d
paper_authors.avatarUrl: /avatars/1f2db6194e24602a1e254e31ba609ee3.svg
paper_authors.fullname: Heinrich Jiang
paper_authors.type: user
paper_authors.user: heinrichjiang
paper_authors._id: 65b1e70254fa188a2fbf4854
paper_authors.name: Afshin Rostamizadeh
paper_authors._id: 65b1e70254fa188a2fbf4855
paper_authors.name: Ayan Chakrabarti
paper_authors._id: 65b1e70254fa188a2fbf4856
paper_authors.name: Giulia DeSalvo
paper_authors._id: 65b1e70254fa188a2fbf4857
paper_authors.name: Jean-François Kagy
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-01-25T04:43:46.730Z
paper_authors._id: 646282962538819c729edca1
paper_authors.avatarUrl: /avatars/89888fef14a957b89333c6f1cc6fab23.svg
paper_authors.fullname: JF Kagy
paper_authors.type: user
paper_authors.user: jfkagy
paper_authors._id: 65b1e70254fa188a2fbf4858
paper_authors.name: Lazaros Karydas
paper_authors._id: 65b1e70254fa188a2fbf4859
paper_authors.name: Gui Citovsky
paper_authors._id: 65b1e70254fa188a2fbf485a
paper_authors.name: Sanjiv Kumar
paper_title: SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced
    Token Detection
paper_summary: Pre-training large language models is known to be extremely resource
  intensive and often times inefficient, under-utilizing the information
  encapsulated in the training text sequences. In this paper, we present SpacTor,
  a new training procedure consisting of (1) a hybrid objective combining span
  corruption (SC) and token replacement detection (RTD), and (2) a two-stage
  curriculum that optimizes the hybrid objective over the initial tau
  iterations, then transitions to standard SC loss. We show empirically that the
  effectiveness of the hybrid objective is tied to the two-stage pre-training
  schedule, and provide extensive analysis on why this is the case. In our
  experiments with encoder-decoder architectures (T5) on a variety of NLP tasks,
  SpacTor-T5 yields the same downstream performance as standard SC pre-training,
  while enabling a 50% reduction in pre-training iterations and 40% reduction in
  total FLOPs. Alternatively, given the same amount of computing budget, we find
  that SpacTor results in significantly improved downstream benchmark
  performance.
paper_ai_summary: SpacTor, a hybrid training procedure combining span corruption and token replacement detection with a two-stage curriculum, enhances pre-training efficiency and performance for large language models.
paper_ai_keywords: span corruption token replacement detection two-stage curriculum encoder-decoder architectures T5 downstream performance pre-training FLOPs