So if I understand you correctly, you minibatch s and a in line 4 and 7 but take the time steps individually? But then you might be able to just take a single w and v by summing the terms in delta terms in 13 + 14 and the sum the log before the differentiation.
If you combine several timesteps, you’d get slower updates of things, I would not know if that is too much…