SKYHOWIE25:

non-empty torch.cuda.ByteTensor is ambiguous


Hmm…I am very new to pytorch so I am still experimenting as well, so I am not 100% sure and correct me please if I am wrong 
But first, the output of criterion/loss function is autograd Variable, since it can be used like

loss = criterion(…)
loss.backward()

And looking at the scheduler class, the scheduler.step() function seems to be accepting a number(like a float), since it looks like it is doing a comparison with some threshold values.
So, when you are adding loss_cls[i], try loss_cls[i].data[0] instead, because variable.data is a tensor and variable.grad is another variable for computing gradients.
You might want to try printing out the output of criterion and check out what type it is and what it is composed of, etc…
Also, maybe you should average out the loss value?