22

8163264LoRA r64.064.264.464.664.865.0RougeLbits4Parameters Dataset
7B
7B
7B
7B
13B
13B
13B
13B
33B
33B
33B
33B
65B
65B
65B
65B

All
OASST1
HH-RLHF
Longform
All
OASST1
HH-RLHF
Longform
All
OASST1
HH-RLHF
Longform
All
OASST1
HH-RLHF
Longform

Batch size
16
16
16
16
16
16
16
16
32
16
32
32
64
16
64
32

LR
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4
1e-4

Steps
10000
1875
10000
4000
10000
1875
10000
4000
5000
1875
5000
2343
2500
1875
2500
2343

Source Length Target Length
128
512
768
1024
128
512
768
1024
128
512
768
1024
128
512
768
1024

384
-
-
512
384
-
-
512
384
-
-
512
384
-
-
512

Table 9: Training hyperparameters for QLORA finetuning on different datasets and across model sizes.
QLORA