So when I get the raw values of the feature maps in the test step, I need to convert them to probabilities applying the annealing and then softmax(), but the problem is that there are negative values and this break the log() of annealing.

First, you run the output through a softmax, and then you do the annealing. The point of the annealing is to take the predicted color distribution, which is unlikely to be sharp, and remap it into one with stronger peaks.