Could you post your confusion matrix?
Is the model ignoring the minority classes?
A minor side note: Instead of np.unique and np.histogram you can directly get the class counts with:
_, counts = np.unique(labels_t, return_counts=True)