I believe just labelling them 0,1,2,3 is the typical approach.
What do you use for the last model layer, and what loss function do you use?
In the literature models for classification typically use a softmax layer as the last layer together with cross-entropy loss. In PyTorch this can be achieved in two ways.

Use nn.LogSoftmax for the last layer together with nn.NLLLoss. But, in this case you must transform your target values into one-hot format. Label 0 would be represented by the vector [1,0,0,0], label 1 would be represented by [0,1,0,0], etc…
Simply use nn.CrossEntropyLoss without the nn.LogSoftmax layer and provide target values as 0, 1, 2, 3.

If that is correct, then the next thing to try is to reduce the learning rate. Though that might simply delay the loss becoming NaN.
If that fails then I wouldn’t know what to suggest without seeing any code.