Sriharsha-hatwar:

I am quite unsure how scaling the loss benefits this.


Accumulating the scaled loss (loss divided by the accumulation steps) would simulate training using a bigger batch size.
E.g. you could perform 10 iterations with a batch size of 1 and accumulate the gradients created by the iter_loss/10, which would create the same gradients as training with a batch size of 10.
This is often used, if you would be running out of memory, but the training would generally benefit from a larger batch size.
Here is a simple code snippet:
lin = nn.Linear(1, 1)

x = torch.randn(10, 1)
y = torch.randn(10, 1)
criterion = nn.MSELoss()

# gradient accumulation approach
for i in range(x.size(0)):
    x_ = x[i:i+1]
    y_ = y[i:i+1]
    out = lin(x_)
    loss = criterion(out, y_)
    loss = loss / x.size(0)
    loss.backward()

print(lin.weight.grad, lin.bias.grad)
> tensor([[-0.9022]]) tensor([0.5286])

# "large batch" approach
lin.zero_grad()
out = lin(x)
loss = criterion(out, y)
loss.backward()

print(lin.weight.grad, lin.bias.grad)
> tensor([[-0.9022]]) tensor([0.5286])

This might not fit your use case and you could thus remove the scaling step.