Stapo:

I understand that adjusting the running stats during evaluation would be a data leak, especially in the case of sequential data where the batch would essentially run calculations on future datapoints.


I think your understanding is correct and it often comes to the question how this model would be deployed into a real-world use case. I.e. I assume you won’t have future data points when the model is deployed, so you also shouldn’t use it during validation.



 Stapo:

I solved this by using batches that have items ordered by time and I only evaluate the last item and therefore the running stats were calculated based on previous data only.


That sounds reasonable.



 Stapo:

I set the batch normalization layers’ momentum to 1 (so that it calculates the means and variances only from the current batch) and evaluate only the last item of the batch.


Alternatively, you could also use track_running_stats=False, which would then calculate the mean and std always from the input activation.



 Stapo:

Did I take the right approach?


I think it boils down again to the test use case. If you are not leaking any future data (or any other data you wouldn’t have), your approach could be fine.