You’re right, when using random seed, result is really unpredictable - sometimes sum-of-grads converges, sometimes - grad-of-sums. So, it must be some weird combination of precision erros and weight initialization causing this effect on my fixed seed…
BTW, either this environment turns out to be much harder for RL than I expected or I have some sneaky bug here. I developed “snake”-like sim - 40x40 squares field with N random wall blocks (and walls on the border), M “apples” and 3-segment snake, controlled by neural net. Every time step it receives 3x8 vectors of distances in 8 directions to nearest wall, “apple”, and self segment (24 distances total). And it can not properly learn, even when I disable grow-on-eating.
Best result I’ve got so far - snake learns to avoid walls, but it is not crazy about “apples”. When I add 25th input to net, representing “satiation” (init it with value 100, every apple adds +100 of it, every step decrements it), training fails completely.
Well, not completely - if I enable “die-on-satiation-0”, then it fails. If I just penalize net with reward -1 for each step on satiation=0, it has amazing effect: while still learning to avoid walls (most death caused by collision), total reward slowly rises. But when it learns to live more than 100 iterations, it begin to receive enormous penalties for “starvations” and total reward drops to negative values. And again - this all happens with disabled grow-on-eating!

(blue is iterations till death, orange is total reward, graphs diverge at value about 100, when snake learns to live long anough to starve)
It is really confuisng and I am still trying to figure out what should I tune in such cases (this RL task seemed very easy to me, I supposed even non-recurrent net should learn optimal behavior in ~1000-5000 episodes…)
I am using the same actor-critic code from pytorch actor-critic example, so there should not be any bugs there.