Exactly, during the evaluation you often pass single images through your net and don’t want to update any parameters or buffers.
The running stats as well as beta and gamma are just applied then.
Same goes for Dropout. During training the units are dropped randomly depending on the specified drop probability. During evaluation all units are used.