Hi @ptrblck, sorry for the mixing of concepts. Yes this question was targeted to check on scaling of loss when using gradient accumulation. Currently this is what I am using with Pytorch’s 1.6 amp package.
                import torch.cuda.amp as amp_py

                scaler = amp_py.GradScaler()
                with amp_py.autocast():
                    outputs = model(images, target_res)
                    loss = outputs['loss']
                scaler.scale(loss).backward()
                train_final_loss.update(loss.item(), batch_size) # An instance of averagemeter class that calculates the average on fly
                if (batch_index + 1) % Config.ACCUMULATE == 0: 
                    scaler.step(optimizer)
                    scaler.update()
                    optimizer.zero_grad()

and this seems to training well for now.
Can you please elaborate this :
If I’m not mistaken, you would divide the loss by the step size to accumulate the gradients in such a way, which would keep the gradient magnitude as it would be using the “bigger batch size”.

from what I understand is that you are not immediately using the gradients to update the training procedure rather you are adding the gradients for steps and after it reaches a threshold you are making the optimiser to update the weights. I am quite unsure how scaling the loss benefits this.