Looks like the problem is coming from me. r was a LongTensor in my case. Casting it to double seems to fix the issue.
Still, I do not understand how rewards are generated as in the reinforce.py example that @smth  earlier mentioned. In the gym example, they seem to the the output of the env.step() function. In my case, I am extracting the last layer of a convnet, sampling from this layer to generate the input to a regressor network. In order to call backward on the stochastic input, I have learnt I have to compute the rewards by calling .reinforce(). I am not sure where the rewards should be populated in my framework. For now, I am doing something like this:
    R, rewards = 0,  []
    for r in policy.rewards[::-1]:   #my policy.rewards is initially an empty list as in the example
        R = r + args.gamma * R
        rewards.insert(0, R)
    rewards = torch.Tensor(rewards)
    #...   ...  ...   ....  ...

From Ronald Williams paper 1, the REINFORCE algorithm is given by:
\Delta w_{ij} = \alpha_{ij} (r - b_{ij} ) e_{ij}

When we call the StochasticVariable.reinforce(someTensor) function in pytorch, what is going on under the hood? How are the reinforcement baseline b_{ij}, characteristic eligibility, e_{ij} and the learning rate factor, \alpha_{ij} computed?
I would like to know to be sure I am doing the correct thing. Sorry for my troubles but thanks for your help so far. I seem to be getting there.