Hi,
I’m also very curious about this question and I do some experiments.
Set-ups:
RTX3090, Ubuntu 20.04, Nvidia Driver 460.32.03, Cuda 11.1
PyTorch 1.7.1, transformers 4.3.2 (huggingface)
Model: Roberta-large (RobertaForMultipleChoice)
Goal: real batch size (16) = effective batch size (bs_gpu=8, gradient accumulation steps=2)?
Fix all random seeds and all hyperparameters
To my knowledge, the Roberta model uses LayerNorm instead of BatchNorm,
which does not have running statistics.
However, to make sure everything is all right,
I used model.eval() even in training and set the weight and bias in each LayerNorm layer to requires_grad=False. I also disabled gradient norm clipping.
==============================================================================
Results:
Orange line: real batch size; Blue line: effective batch size (using gradient_accumulation)
I only show the plot before the 1st epoch ends in case that the last batch matters
The x-axis is the #global steps (= effective steps)
Fig1: torch.abs(classifier.weight.grad).sum()
Fig2: torch.abs(output).sum() given the same input
image512×744 96.2 KB
There might have some bugs in my code,
if not, my observation is that effective batch size != real batch size,
and the difference becomes very significant after a few steps.
Could it be gradient_accumulation causes some numerical issues and leads to error propagation?
BTW I make sure the plots are the same given the same run using real_batch_size (my code is reproducible)
I wonder if my experiments and suggestion are right?
Thanks