The optimizers should update the parameter inplace as shown here:
p = nn.Parameter(torch.ones(10).pin_memory())
print(p.is_pinned())
optimizer = torch.optim.Adam([p], lr=1.)
p.grad = torch.ones_like(p)
print(p.is_pinned())
optimizer.step()
print(p.is_pinned())

What is your use case to update the parameters on the CPU?