AdamW

128
2

250,000
Linear

5.00E-06

5.00E-04

1.00E-04

1.6E-03

1.00E-04

2.00E-04

Table 12: The training hyperparameters used for different GPT-3 adaption methods. We use the
same hyperparameters for all datasets after tuning learning rate.
LORA