title: Decreased accuracy when converting keras code to pytorch
body: <p>I am a beginner in Pytorch and I am trying to convert a keras code to Pytorch for classification task.
  The results are different for me in two modes with the same settings. Accuracy is reduced by 5% and the Loss is increases from 0.2 to 4.5. Can anyone tell me if I converted the code correctly to Pytorch or not? Thank you so much.
  IMG_HEIGHT=48 ,IMG_WIDTH=48 , channels=3, num_class=164</p>
  <pre><code>model = keras.models.Sequential([    
  keras.layers.Conv2D(filters=16, kernel_size=(3,3), activation='relu', input_shape=(IMG_HEIGHT,IMG_WIDTH,channels)),
  keras.layers.Conv2D(filters=32, kernel_size=(3,3), activation='relu'),
  keras.layers.MaxPool2D(pool_size=(2, 2)),
  keras.layers.BatchNormalization(axis=-1),
  
  keras.layers.Conv2D(filters=64, kernel_size=(3,3), activation='relu'),
  keras.layers.Conv2D(filters=128, kernel_size=(3,3), activation='relu'),
  keras.layers.MaxPool2D(pool_size=(2, 2)),
  keras.layers.BatchNormalization(axis=-1),
  
  keras.layers.Flatten(),
  keras.layers.Dense(512, activation='relu'),
  keras.layers.BatchNormalization(),
  keras.layers.Dropout(rate=0.5),
  
  keras.layers.Dense(164, activation='softmax')])
  </code></pre>
  <p>pytorch :</p>
  <pre><code>class convnet(nn.Module):
  def __init__(self, num_class):
      super(convnet, self).__init__()
      self.Conv2d_1 = nn.Conv2d(3, 16, (3,3))
      self.relu_1 = nn.ReLU()
      self.Conv2d_2 = nn.Conv2d(16, 32, (3,3))
      self.relu_2 = nn.ReLU()
      self.maxpool_1 = nn.MaxPool2d(kernel_size=2, stride=2)
      self.batch_1 = nn.BatchNorm2d(32)
  
      self.Conv2d_3 = nn.Conv2d(32, 64, (3,3))
      self.relu_3 = nn.ReLU()
      self.Conv2d_4 = nn.Conv2d(64, 128, (3,3))
      self.relu_4 = nn.ReLU()
      self.maxpool_2 = nn.MaxPool2d(kernel_size=2, stride=2)
      self.batch_2 = nn.BatchNorm2d(128)        
  
      self.fc1 = nn.Linear(10368, 1024)  
      self.relu_5 = nn.ReLU()
      self.batch_3 = nn.BatchNorm1d(1024)      
      self.dropout_1 = nn.Dropout(p=0.5)
      self.fc2 = nn.Linear(1024,num_class)
      self.Softmax_1 = nn.Softmax()
  
  def forward(self, x):
  
      y = self.Conv2d_1(x)
      y = self.relu_1(y)
      y = self.Conv2d_2(y)
      y = self.relu_2(y)
      y = self.maxpool_1(y)
      y = self.batch_1(y)
  
      y = self.Conv2d_3(y)
      y = self.relu_3(y)
      y = self.Conv2d_4(y)
      y = self.relu_4(y)
      y = self.maxpool_2(y)
      y = self.batch_2(y)
  
      y = y.view(y.size(0), -1)
      y = self.fc1(y)
      y = self.relu_5(y)
      y = self.batch_3(y)
      y = self.dropout_1(y)
      y = self.fc2(y)
      y = self.Softmax_1(y)
  
      return y
  </code></pre>