I eventually figured it out – originally, my forward method was outputting the FloatTensor, and I was only storing the contents. When backpropagating the loss, I would wrap this in a variable, and use backward(), and the reward would increase, but nowhere near what it should have. I fixed it by returning and storing the variable (rather than the contents) and it worked as expected.