Pre-training Language Model as a Multi-perspective Course Learner
ELECTRA, the generator-discriminator pre-training framework, has achieved
impressive semantic construction capability among various downstream tasks.
Despite the convincing performance, ELECTRA still faces the challenges of
monotonous training and deficient interaction. Generator with only masked
language modeling (MLM) leads to biased learning and label imbalance for
discriminator, decreasing learning efficiency; no explicit feedback loop from
discriminator to generator results in the chasm between these two components,
underutilizing the course learning. In this study, a multi-perspective course
learning (MCL) method is proposed to fetch a many degrees and visual angles for
sample-efficient pre-training, and to fully leverage the relationship between
generator and discriminator. Concretely, three self-supervision courses are
designed to alleviate inherent flaws of MLM and balance the label in a
multi-perspective way. Besides, two self-correction courses are proposed to
bridge the chasm between the two encoders by creating a "correction notebook"
for secondary-supervision. Moreover, a course soups trial is conducted to solve
the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained
model. Experimental results show that our method significantly improves
ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on
GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style
models under the same settings. The pre-trained MCL model is available at
https://huggingface.co/McmanusChen/MCL-base.
6459a52d0aba070266e36b0c
Beiduo Chen
admin_assigned
2023-06-02T14:12:13.708Z
645474d788c97c8796dcb3d8
/avatars/dfab5c337c2c3d9c44149738f2e09517.svg
Beiduo CHen
user
McmanusChen
6459a52d0aba070266e36b0d
Shaohan Huang
admin_assigned
2023-06-02T14:12:37.828Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
6459a52d0aba070266e36b0e
Zihan Zhang
6459a52d0aba070266e36b0f
Wu Guo
6459a52d0aba070266e36b10
Zhenhua Ling
6459a52d0aba070266e36b11
Haizhen Huang
admin_assigned
2023-06-02T14:14:22.029Z
6302e8e57e137e3d6b3f5a06
/avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
Huang
user
Haizhen
6459a52d0aba070266e36b12
Furu Wei
admin_assigned
2023-06-02T14:14:35.264Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
6459a52d0aba070266e36b13
Weiwei Deng
admin_assigned
2023-06-02T14:14:53.090Z
642da49dc1adada5b345c72b
/avatars/5fa8d345a7248246454444999756105b.svg
deng
user
weiweideng
6459a52d0aba070266e36b14
Qi Zhang
Pre-training Language Model as a Multi-perspective Course Learner
ELECTRA, the generator-discriminator pre-training framework, has achieved
impressive semantic construction capability among various downstream tasks.
Despite the convincing performance, ELECTRA still faces the challenges of
monotonous training and deficient interaction. Generator with only masked
language modeling (MLM) leads to biased learning and label imbalance for
discriminator, decreasing learning efficiency; no explicit feedback loop from
discriminator to generator results in the chasm between these two components,
underutilizing the course learning. In this study, a multi-perspective course
learning (MCL) method is proposed to fetch a many degrees and visual angles for
sample-efficient pre-training, and to fully leverage the relationship between
generator and discriminator. Concretely, three self-supervision courses are
designed to alleviate inherent flaws of MLM and balance the label in a
multi-perspective way. Besides, two self-correction courses are proposed to
bridge the chasm between the two encoders by creating a "correction notebook"
for secondary-supervision. Moreover, a course soups trial is conducted to solve
the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained
model. Experimental results show that our method significantly improves
ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on
GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style
models under the same settings. The pre-trained MCL model is available at
https://huggingface.co/McmanusChen/MCL-base.
A multi-perspective course learning (MCL) method enhances ELECTRA's pre-training by designing self-supervision and self-correction courses, improving its performance on GLUE and SQuAD 2.0 benchmarks.
ELECTRA generator-discriminator pre-training masked language modeling (MLM) multi-perspective course learning (MCL) self-supervision self-correction pre-trained model GLUE SQuAD 2.0