title: Pre-training Language Model as a Multi-perspective Course Learner
summary: ELECTRA, the generator-discriminator pre-training framework, has achieved
  impressive semantic construction capability among various downstream tasks.
  Despite the convincing performance, ELECTRA still faces the challenges of
  monotonous training and deficient interaction. Generator with only masked
  language modeling (MLM) leads to biased learning and label imbalance for
  discriminator, decreasing learning efficiency; no explicit feedback loop from
  discriminator to generator results in the chasm between these two components,
  underutilizing the course learning. In this study, a multi-perspective course
  learning (MCL) method is proposed to fetch a many degrees and visual angles for
  sample-efficient pre-training, and to fully leverage the relationship between
  generator and discriminator. Concretely, three self-supervision courses are
  designed to alleviate inherent flaws of MLM and balance the label in a
  multi-perspective way. Besides, two self-correction courses are proposed to
  bridge the chasm between the two encoders by creating a "correction notebook"
  for secondary-supervision. Moreover, a course soups trial is conducted to solve
  the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained
  model. Experimental results show that our method significantly improves
  ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on
  GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style
  models under the same settings. The pre-trained MCL model is available at
  https://huggingface.co/McmanusChen/MCL-base.
paper_authors._id: 6459a52d0aba070266e36b0c
paper_authors.name: Beiduo Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:12:13.708Z
paper_authors._id: 645474d788c97c8796dcb3d8
paper_authors.avatarUrl: /avatars/dfab5c337c2c3d9c44149738f2e09517.svg
paper_authors.fullname: Beiduo CHen
paper_authors.type: user
paper_authors.user: McmanusChen
paper_authors._id: 6459a52d0aba070266e36b0d
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:12:37.828Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 6459a52d0aba070266e36b0e
paper_authors.name: Zihan Zhang
paper_authors._id: 6459a52d0aba070266e36b0f
paper_authors.name: Wu Guo
paper_authors._id: 6459a52d0aba070266e36b10
paper_authors.name: Zhenhua Ling
paper_authors._id: 6459a52d0aba070266e36b11
paper_authors.name: Haizhen Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:14:22.029Z
paper_authors._id: 6302e8e57e137e3d6b3f5a06
paper_authors.avatarUrl: /avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Haizhen
paper_authors._id: 6459a52d0aba070266e36b12
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:14:35.264Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 6459a52d0aba070266e36b13
paper_authors.name: Weiwei Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:14:53.090Z
paper_authors._id: 642da49dc1adada5b345c72b
paper_authors.avatarUrl: /avatars/5fa8d345a7248246454444999756105b.svg
paper_authors.fullname: deng
paper_authors.type: user
paper_authors.user: weiweideng
paper_authors._id: 6459a52d0aba070266e36b14
paper_authors.name: Qi Zhang
paper_title: Pre-training Language Model as a Multi-perspective Course Learner
paper_summary: ELECTRA, the generator-discriminator pre-training framework, has achieved
  impressive semantic construction capability among various downstream tasks.
  Despite the convincing performance, ELECTRA still faces the challenges of
  monotonous training and deficient interaction. Generator with only masked
  language modeling (MLM) leads to biased learning and label imbalance for
  discriminator, decreasing learning efficiency; no explicit feedback loop from
  discriminator to generator results in the chasm between these two components,
  underutilizing the course learning. In this study, a multi-perspective course
  learning (MCL) method is proposed to fetch a many degrees and visual angles for
  sample-efficient pre-training, and to fully leverage the relationship between
  generator and discriminator. Concretely, three self-supervision courses are
  designed to alleviate inherent flaws of MLM and balance the label in a
  multi-perspective way. Besides, two self-correction courses are proposed to
  bridge the chasm between the two encoders by creating a "correction notebook"
  for secondary-supervision. Moreover, a course soups trial is conducted to solve
  the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained
  model. Experimental results show that our method significantly improves
  ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on
  GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style
  models under the same settings. The pre-trained MCL model is available at
  https://huggingface.co/McmanusChen/MCL-base.
paper_ai_keywords: generator-discriminator pre-training framework semantic construction masked language modeling (MLM) label imbalance learning efficiency feedback loop multi-perspective course learning (MCL) self-supervision courses self-correction courses correction notebook secondary-supervision course soups trial Pearson-of-war dynamics GLUE benchmark SQuAD 2.0 benchmark pre-trained model