I found an issue using LBFGS optimizer.
I need to return some tensors computed inside closure along with the loss value. Adam and SGD work well but LBFGS can’t handle the extra tensors and returns:
TypeError: float() argument must be a string or a number, not ‘tuple’
Toy example:
import torch
X=torch.tensor([1,2,3,4], dtype=torch.float32)
Y=torch.tensor([2,4,6,8], dtype=torch.float32)
w=torch.tensor(0.0, dtype=torch.float32, requires_grad=True)

# model prediction
def forward(x):
    return w*x

print(f'Prediction before training: f(5) = {forward(5):.3f}')

# training
lr = 0.1
nit=100

loss= torch.nn.MSELoss()
#optimizer=torch.optim.SGD([w],lr=lr)
#optimizer=torch.optim.Adam([w],lr=lr)
optimizer=torch.optim.LBFGS([w],lr=lr)

def closure():

    optimizer.zero_grad()
    y_pred=forward(X)
    myloss=loss(Y,y_pred)
    myloss.backward()
    #print('ypred is:',pred)
    return myloss,y_pred

for epoch in range(nit):

    l,pred=optimizer.step(closure)
    
    if epoch %10 ==0:
        print(f'epoch {epoch+1}: w = {w:.3f},. loss = {l:.8f}')

print(f'Prediction after training: f(5) = {forward(5):.3f}')
print('ypred:',pred)

Example adapted form:
https://github.com/python-engineer/pytorchTutorial/blob/master 1
File:06_1_loss_and_optimizer.py