Hmmm, so, you got me worried that I was using the wrong loss function 
So, I worked through it a bit https://github.com/hughperkins/pub-prototyping/blob/master/maths/sarsa_updates.ipynb 47
… and I’m fairly sure of two things:

should be MSE loss. Though there might be some issue with my reasoning above?, and
to directly backprop delta without needing a loss function, we can do simply:

q.backward(-delta)
Thoughts?