rate of 1 × 10−4 with 500 warmup steps and a cosine scheduler, and the same parameters for weight decay and Adam
optimizer as the pre-training phase. The Adam epsilon is set to 1 × 10−5. We pre-train for one epoch, and instruct-tune
for a total of 10 epochs.
For DocLLM-7B, pre-training involves a learning rate of 3 × 10−4 with 1,000 warmup steps and cosine scheduler,
weight decay of 0.1, and Adam optimizer with β1 = 0.9, β2 = 0.95. Instruction tuning uses a learning rate of 1 × 10−4
with 500 warmup steps and a cosine scheduler, weight decay of 0.1, and Adam optimizer with β1 = 0.9, β2 = 0.95.
Adam epsilon is set at 1 × 10−6. We conduct one epoch of pre-training, followed by three epochs of instruct-tuning,
considering available computing resources.
The maximum sequence length, or context length, is consistently set to 1,024 for both versions during the entire training
process. The DocLLM-7B models are trained with 16-bit mixed precision on 8 24GB A10g GPUs using fully sharded
DOCLLM