Hi, @jekbradbury thanks a lot!
I tired conversion to a torch tensor but couldn’t get it to work - I’ll try again thought?
What seems to help a little is, changing the code to
    for t in reversed(range(len(rewards))):
        R = torch.mul(R, args.gamma)  
        R = torch.add(R, rewards[t])
        advantage = R - values[t]
        value_loss = value_loss + advantage.pow(2)

Now I get the error,
  File "main_single.py", line 185, in <module>
    (policy_loss + 0.5 * value_loss).backward()
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/autograd/variable.py", line 158, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/autograd/stochastic_function.py", line 13, in _do_backward
    raise RuntimeError("differentiating stochastic functions requires "
RuntimeError: differentiating stochastic functions requires providing a reward

Which is perhaps a little bit better than before? I think gym environments are a bit strange?