Your approach might work, if you are using a simple optimizer, e.g. SGD without momentum.
When you are using momentum or an optimizer with running estimates, the reversed step might not work out of the box and you might need to look into the applied formula to check, how to revert it.
Here is a small example using SGD:
torch.manual_seed(2809)

# Set model to eval to prevent batchnorm and dropout layers of changing the output
model = models.resnet50().eval()
x = torch.randn(2, 3, 224, 224)
target = torch.randint(0, 1000, (2,))

optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# Dummy update steps
out = model(x)
loss = criterion(out, target)
loss.backward()
print('Initial loss ', loss.item())
optimizer.step()
fc0 = model.fc.weight.detach().clone()

# Get updated loss
out = model(x)
loss = criterion(out, target)
print('Updated loss ', loss.item())
fc1 = model.fc.weight.detach().clone()

# Use negative lr
optimizer.param_groups[0]['lr'] = -1. * optimizer.param_groups[0]['lr']
optimizer.step()
out = model(x)
loss = criterion(out, target)
print('Reverted loss ', loss.item())
fc2 = model.fc.weight.detach().clone()

optimizer.step() should not increase the memory usage. Are you sure this line causes the OOM issue?
Usually you would run out of memory, if you are storing some tensors, which are not detached from the computation graph.