I’m not sure if I understand the problem correctly.
As far as I’ve understood it, you have clean training data and mixed-up Val data. Now your validation accuracy does represent the Val data classes, i.e. 50% accuracy for 50% wrong images.
Could you explain your issue a bit more?