Ah, sorry I misread the code. When doing gradient accumulation, one step is one backward pass, so appends every gradient_accumulation_steps examples. So you need to lower your logging_steps/eval_steps/save_steps to see something happening.