Yeah from what I understand, you will get better accuracy if your class sizes are close to equal. I think varying data augmentation could help or culling from the largest classes or using synthetic data to fill in some of the smaller sizes.