The multiple initializations are expected in case you are using num_workers>0, since each worker would create a Dataset copy and would thus also re-initialize the neural network to preprocess the data.
Your approach is uncommon and I would guess that the more common use case would be to use the NN to preprocess the data into the DataLoader loop (in the same way the trainable model would be used).