q_sa_next should have gradients stripped. ie call Variable(q_sa_next.data) or q_sa_next.detach().
if it was me, I’d simply call .data on it, to strip out the gradients. And then add Variable into the update line:
update = Variable(alpha * (reward + gamma * q_sa_next) - q_sa

I never quite figured out how to backpropagate update directly though, so I do it like this:
crit = nn.MSELoss()
loss = crit(q_sa, Variable(reward + gamma * q_sa_next))
loss.backward()

(Edit: I guess that to backpropagate update, we’d do something like q_sa.backward(update). But as I say I just lazily do it using crit for now, and never tested this approach)