Solved by vabh in post #2 
              
              
                You should call model.eval() before testing and model.train() before starting a training loop. 
http://pytorch.org/docs/master/nn.html#torch.nn.Module.eval 
You won’t need to see the train attribute manually for every layer.