for inputs in self.train_loader:

                loss = self.loss_fn(sr, hr)
                self.optim.zero_grad()
                loss.backward()
                nn.utils.clip_grad_norm_(self.refiner.parameters(), cfg.clip)
                self.optim.step()

This is the core of your code. If you don’t call zero grad and step, you will accumulate gradients corresponding to the different iterations. Once you have reached the amount of iterations to accumulate you just need to call step and later on zero grad to reset gradients. In short, zero grad sets all gradients on zero and step upgrades model parameters with whatever is stored on gradients.
According to your answer your problem is how do you define “iteration” in this case. Here you can consider an iteration as you were doing or you can consider that iteration is the amount of times you upgrade parameters. The second option is the conventional one.