That is the problematic batch. I printed out every batch to see whether any changes are happening right before the problematic batch or if any unusual happens. But thanks for the tip!
But nothing unusual is happening. The Q_eval is constantly 32 x 200 and both index is constantly a range between 0 to 31 and action batch is always between 0 and 199. Though I have no idea why the error occurs.
And to comment on your idea. it doesn’t really speed up the process of debugging if I only print the faulty batch. This still occurs after around 1 hour which is what bothers me. But nothing changes in the code after 1 hour which makes this question to an open debate…