You approaches sounds reasonable and it’s hard to tell what could work.
I would generally recommend to check the complete data loading and preprocessing pipeline.
Sometimes bugs might have been added, such as mixed up labels etc., which are really nasty.
Once this is done, you could try to increase the augmentation for the training set and hope that it would better capture the underlying data distribution.
If that all doesn’t help, run a quick test by swapping the validation Dataset for the training set.
This should give you a good validation loss. If that’s not the case, recheck the validation pipeline, as there might be an issue.