No, actually, softmax normalize logits between [0, 1] in the way that sum of all values adds up to 1, So, it can be interpreted as probability. And cross entropy does this internally.
For instance, a output of nn.Linear can be like this for 2 classes:
output = [[-2.3, 8.46],
          [-.56, 3.24], 
                   ...]

And after applying softmax:
output = [[0.1, 0.9],
          [0.3, 0.7],
                 ...]

edit: explanation was based on another library and partly wrong.
Thanks @timesler