No worries I appreciate your time. Just to make sure I understand, is the order of my steps wrong? Right now I have
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

And I do this for every batch, is this not correct? Not sure what you mean by “shouldn’t do zero_grad() after forward until all processing is done.”