You need three separate BatchNorm layers, because BatchNorm also keeps track of running_mean and running_var, batch statistics which are used at test time. These statistics will be quite different at x3. x4 and x5.