Sounds like BatchNorm–when you call .train() on your net, you put all of its child modules into training mode, including the BatchNorm modules. When BatchNorm is in training mode, every time you run a forward pass, it is updating its “running mean” and “running variance” parameters on its own (i.e. no interaction with SGD or any user-defined optimizer) which will definitely have an effect on your observed output. I’m not sure if this holds true when the Variable being passed through is volatile, but given what you’re seeing I’m going to guess that it does.
If you don’t want these updates to happen, put the BatchNorm modules in inference mode with model.eval(), or just set the specific BatchNorm modules to inference mode if you need the other modules in training mode (i.e. loop through all modules and have a conditional that checks if a module is nn.BatchNorm2d, and calls .eval() on it if it is).
Or you could probably set the momentum term of each BatchNorm module to 0–IIRC, that’s the term that controls the update rate of the running means and variances, so if you still want to use the per-batch means and variances but you don’t want to update the running statistics, setting momentum to 0 should allow you to do that.