Hi,
If I use T5 language model, there isn’t any batch-norm layer inside, should I use averaged loss then?
Or I can just use loss.backward() on every iteration and opt.step() opt.zero_grad() when iteration % accumulation_steps == 0 ?