i think that it’s better to train model for more epochs. also you can use larger lr.
your loss is still to high, maybe training for a bit more give you a better result,
you can use weight decay to have more generalization.
i found these topics, i hope it helps you.

Expected calibration error (ECE) is a metric that compares neural network model output pseudo-probabilities to model accuracies . ECE values can be used to calibrate (adjust) a neural network model so that output pseudo-probabilities more closely match actual probabilities of a correct prediction.


Implicit model calibration:


By artificially softening the targets, label smoothing prevents the network from becoming overconfident. But does it improve the calibration of the model by making the confidence of its predictions more accurately represent their accuracy? In this section, we seek to answer this question. Guo et al. [15] have shown that modern neural networks are poorly calibrated and over-confident despite having better performance than better calibrated networks from the past. To measure calibration, the authors computed the estimated expected calibration error (ECE). They demonstrated that a simple post-processing step, temperature scaling, can reduce ECE and calibrate the network. Temperature scaling consists in multiplying the logits by a scalar before applying the softmax operator. Here, we show that label smoothing also reduces ECE and can be used to calibrate a network without the need for temperature scaling.

and i think you should check class probability of each sample not class accuracy