Thanks. It was because of softmax.
What i have learned as a newbie in pytorch is as follow:
If I have L1 and L2 as loss values for two different output nodes, I can simply do L = mean(L1,L2) and then  L.backward(). But the trick is, since, in reinforcement learning (RL), the target value is not calculated based on NN output nodes, first we need to associate loss/reward to output nodes. To do so, in this example, doing (L1+ys[0][0].detach() - ys[0][0]) would keep the value of error still L1 but would make it connect to “ys”, so doing backward() would indeed adjust weights.
Similar approach is also taken in classic RL. Once we calculate reward R, it needs to be applied to the probability of chosen action. So, if we have output values as a tensor ys = [y1, y2], and assuming that agent has chosen the first output in this episode, i.e. [1,0], the loss function would be [y1,y2].*[1,0].*R, which connects R (which doesn’t have gradients) to ys (which has grads).
Hope this is helpful