this just reflects that gradients and optimizer steps from training become smaller. and the “noise” is obviously from suboptimal stochastic steps in parameter space w.r.t. validation set.
optimizer tuning would maybe achieve faster training loss convergence, and would affect a validation loss graph indirectly. OTOH, big steps may lead to solutions that generalize worse.