Dataset

20 newsgroups
Crowdﬂower airline
Crowdﬂower corporate messaging
Crowdﬂower disasters
Crowdﬂower economic news relevance
Crowdﬂower emotion
Crowdﬂower global warming
Crowdﬂower political audience
Crowdﬂower political bias
Crowdﬂower political message
Crowdﬂower primary emotions
Crowdﬂower progressive opinion
Crowdﬂower progressive stance
Crowdﬂower US economic performance
Customer complaint database
News aggregator dataset
SMS spam collection
Average
Total number of params
Trained params/task

No BERT
baseline

BERTBASE
Fine-tune
92.8 ± 0.1
83.6 ± 0.3
92.5 ± 0.5
85.3 ± 0.4
82.1 ± 0.0
38.4 ± 0.1
84.2 ± 0.4
80.9 ± 0.3
75.2 ± 0.9
38.9 ± 0.6
36.9 ± 1.6
71.6 ± 0.5
63.8 ± 1.0
75.3 ± 0.1
55.9 ± 0.1
96.3 ± 0.0
99.3 ± 0.2
73.7
17×
100%

BERTBASE
Variable FT
92.8 ± 0.1
84.0 ± 0.1
92.4 ± 0.6
85.3 ± 0.4
78.9 ± 2.8
37.6 ± 0.2
81.9 ± 0.2
80.7 ± 0.8
76.5 ± 0.4
44.9 ± 0.6
38.2 ± 1.0
75.9 ± 1.3
61.5 ± 1.3
76.5 ± 0.4
56.4 ± 0.1
96.5 ± 0.0
99.3 ± 0.2
74.0
9.9×
52.9%
Parameter-Efficient Transfer Learning for NLP