That’s a strange issue, as the probs passed to Categorical are created using F.softmax(action_scores, dim=-1). Could you check if the dimension is set properly for your action_scores? F.softmax should not return negative values.
Also, could you add a print statement of action_states and probs just for the sake of debugging?