Your code isn’t executable and raises an error at:
    loss = -q_vals*batch_log_probs

TypeError: only integer tensors of a single element can be converted to an index

After adding:
batch_log_probs = torch.stack(batch_log_probs)

before the problematic operation and fixing loss/mean() to loss.mean() the code runs fine and doesn’t raise the reported error.