I am estimating age from face images where the number of classes is 101(0…100). I made 3 subgroup where group1(0-15),group2(16-64) and group3(65+). The training & validation accuracy of group1 & 3 is  90% & 75% respectively, but group2 validation performance is about (30%) whereas training accuracy is 90%. Noted that, group2 has almost 40k training images and is almost balanced.  I have tried with different pretrained model(resnet, squeezenet, inception resnet v1), scratch models(resnet,vgg16, ), attention models (Squeeze and Excitation, Block Attention module ), but none of them work fine. I have changed the hyper parameters like (weight decay, learning rate scheduler). Can anyone suggest to me which models can work or how I can solve this problem?
FYI: Group2 belongs 49 classes. I am estimating the real age through classification.