optim.LBFGS will optimize the Variables you pass in on a GPU; there is nothing in the source code 48 that would suggest otherwise.
Let’s say you’re optimizing over the Variables x and a function f(x). Try something like the following:
import torch
from torch.autograd import Variable
from torch.optim import LBFGS

x = Variable(torch.Tensor([100]).cuda(), requires_grad=True)
optimizer = LBFGS([x])
def f(x):
	return (x - 1) * (x - 1)

n_step = 2

for _ in range(0, n_step):
    def closure():
        optimizer.zero_grad()
        output = f(x)
        output.backward()
        return output
    optimizer.step(closure)
    print(x)