Im still stuck with my loss problem. This is the loss over time when i train one timestep at a time. Each track is 3000 timesteps long.
The first image is the loss over 50 tracks (50 * 3000 timesteps). The constant low loss between 100000-130000 is because the same class is chosen again due to poor shuffling.

Here is loss for just two consecutive tracks. Pretty fast the loss falls to acceptable values. The loss is NLL so 0.45 is equal to something like 65% confidence in the correct class.

Any help regarding how to update my model and correctly calculate loss is highly appreciated. Right now i input a timestep. Then calculate loss and do the backpropagation and update weights and biases then input new timestep etc.
Unfortunately its like my network forgets everything at the next track.