can you get the training curves as well? if the training is doing well on all groups then there is an under-representation of that group in the test set . The whole idea is difference in trends between train and test.
Also you do have equal number of samples from all groups?