Indeed, this is far from efficient, and good thinking for the normalization, it slipped my mind!
I’m not sure if this is the right way for your kind of data (I’ve only worked on images), but in general, what you want to do is compute your statistics in advance, on the whole training set (do not include validation or test sets, as this would introduce some bias), and then normalize input data in the dataset (not the data loader!) as it gets called, using your previously computed stats.