I think the oscillating behavior of the validation accuracy might depend on its size. I.e. if the validation dataset is “small”, it would yield noisy estimates.
However, it’s hard to quantify what a “small” dataset size would be and it would be interesting to see your training, validation, and test sizes. As a quick debugging step, you could also swap the training and validation datasets and check how the losses behave.