You’re trying to do classification with a distance based loss. You’d need to output label scores (…,output_size, num_classes), treat them as probabilities (softmax) and use appropriate loss like CrossEntropyLoss. Or do something else, but don’t treat labels as floating point numbers.
NB date embeddings are probably fine, you could also encode them with fourier series coefficients or one-hot encoding, avoiding some backprop.