If I don’t call loss.backwards after optimizer.step(), the GPU usage keeps on increasing.
for step_index in range(8):
    optimizer.step()
    outputs = model(X)
    loss = loss_function(outputs, y)
    loss.backward() // If I comment out, GPU usage increases until OOM error

    if step_index == 1:
        to_load_state = {
                'model': model.state_dict()
        }