Hi Markus!



 CesMak:

My problem now was that I am not sure how to interpret the output if I give an 180x1 input vector to the trained model. I get back the following:
learning_rate = 1e-5
tensor([ 0.0982,  0.0182,  0.0380,  0.0131, -0.1438, -0.1540, -0.1128, -0.1452,
...

What is now the action index I should play -> Taking e.g. 21 did not work (playing moves that are not possible)


As to how to interpret the output, the output of your network means
(as always) what you train it to mean.
You are training a conventional 60-class classifier using cross-entropy
(with categorical class labels) as the loss function.  What does this
mean?
First, you are using CrossEntropyLoss.  This takes raw-score logits
as its predictions (the output of your network).  Internally to
CrossEntropyLoss these are converted to probabilities by (in effect)
passing them through softmax().
Cross-entropy then measures how dissimilar two probability distributions
are.
What are your two probability distributions?  They are the probability
of “0” being the right, move, the probability of “1” being the right
move, and so on.  That is, they are discrete probability distributions
on the space [0, 59] (inclusive).  As probabilities, they are between
0 and 1, and sum to 1, sum{i = 0, 59} (P(i)) = 1.
Your first probability distribution is your prediction, that is, the output of
your network (more precisely, the output of your network after passing
it through softmax()).  Your second distribution is your target.  In this
case your target probability distribution has a special form – exactly
one of the 60 probabilities is 1, with the other 59 being 0.
So you are training your network to output the probability of of each
of the 60 moves being the right move.  The simplest rule will be to
play the move that has the largest predicted probability of being right.
(Again, your network outputs logits, but the larger (more positive) the
logit, the larger the probability, and the smaller (more negative) the
logit, the smaller the probability, so choosing the move with the largest
logit is the same as choosing the move with the largest probability.)
Again, the output of your network means what you train it to mean,
and you’ve trained yours (by using CrossEntropyLoss) to output
the predicted probability of each of the 60 moves being right.
Now, as a practical matter, in order to play the game, you know that
only some moves are actually possible.  Your network – not being
perfect – may sometimes predict an impossible move.  It’s perfectly
reasonable to play the possible move with the largest predicted
logit (probability), rather than playing an impossible move with a
larger logit.  Here, you would be taking the output of your (imperfect)
network and combining it with some simple knowledge you have that
your network hasn’t (fully) learned.  In practice, that’s okay.
Best.
K. Frank