Not only the rewards but also the transition probabilities determine your q-values. So please check your transition probabilities.
If you only have 2 states and 4 actions you can also compute your q-function numerically with e.g. value iteration. You can then compare the numerically computed results with the ones of your Pytorch model.