Check also if these classes are imbalanced:

  
      

      TensorFlow
  

  
    

Classification on imbalanced data  |  TensorFlow Core 1



  

  
    
    
  

  


Other then these you can also use some imbalanced loss.