Thank you for your reply! I later realized that I just need the value of the the loss, not all the metadata. so instead of saving loss and outputs, I changed my code as
def closure():
    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, targets)
    loss.backward()
    return loss.detach(), outputs.detach()
loss, outputs = optimizer.step(closure)

That works fine, no out-of-memory error.