What I mean is that optimizer.step() is an inplace operation. So if you try to backward again after doing the optimizer step without re-doing the forward, you will see this error.
Is your code doing something like:
out = model(inp)
first_loss = bar(out, label)
first_loss.backward(retain_graph=True)
opt.step()

opt.zero_grad()
second_loss = baz(out, label)
second_loss.backward()  # This will fail if model has a Linear
# Because the Linear's weight were modified inplace above and
# this backward cannot be done anymore.