So, removing the sigmoid in the output layer worked to discriminate between the classes! My output obviously is no longer on a 0:1 scale it can now become negative and above 1 but I dont think this matters much since taking the maximum of the output will still return the correct prediction on the test data. Could you expand more on how to use the Cross-Entropy Loss without one-hot encoding? I thought that it was necessary for multi-class problems? My current targets are integers ranging from 1:3.