Answers.Body: <p>So batch-normalization helps descent based learning have an easier time traversing the loss manifold, but in your case you use it along with a relu as a final activation is problematic, it means the output is relatively associated with the other samples in the batch.  </p>
  
  <p>Remove that last BN and you get better results, but also understand BN is inherently problematic for this task. Think of DNNs as featurizers, and BN in this case takes out the 2 batch-wide statistics which if they don't align to the initial distribution will cause error, which will lead to error in the output. In theory if BN gets the perfect statistics of the gaussian it should not matter too much, so one thing I tried with your code was remove last BN and increase N to 100,000 while increasing the batch size to 10000 and you see a huge boost in performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-03T18:21:23.000Z
Answers.LastActivityDate: 2019-11-03T18:21:23.000Z
Answers.UserDisplayName: mshlis
Body: <p>I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and   the other without BN.</p>
  
  <p>I find that the model DNN with BN can't predict well.  </p>
  
  <p>The codes is shown as follow:</p>
  
  
  
  <pre><code>import numpy as np
  import scipy.stats
  import matplotlib.pyplot as plt
  from keras.models import Sequential
  from keras.layers import Dense,Dropout,Activation, BatchNormalization
  
  np.random.seed(1)
  
  # generate non-gaussian data
  def generate_data():
      distribution = scipy.stats.gengamma(1, 70, loc=10, scale=100)
      x = distribution.rvs(size=10000)
      # plt.hist(x)
      # plt.show()
      print ('[mean, var, skew, kurtosis]', distribution.stats('mvsk'))
  
      y = np.sin(x) + np.cos(x) + np.sqrt(x)
      plt.hist(y)
      # plt.show()
      # print(y)
      return x ,y 
  
  x, y = generate_data()
  
  x_train = x[:int(len(x)*0.8)]
  y_train = y[:int(len(y)*0.8)]
  x_test = x[int(len(x)*0.8):]
  y_test = y[int(len(y)*0.8):]
  
  
  def DNN(input_dim, output_dim, useBN = True):
      '''
      定义一个DNN model
      '''
      model=Sequential()
  
      model.add(Dense(128,input_dim= input_dim))
      if useBN:
          model.add(BatchNormalization())
      model.add(Activation('tanh'))
      model.add(Dropout(0.5))
  
      model.add(Dense(50))
      if useBN:
          model.add(BatchNormalization())
      model.add(Activation('tanh'))
      model.add(Dropout(0.5))
  
      model.add(Dense(output_dim))
      if useBN:
          model.add(BatchNormalization())
      model.add(Activation('relu'))
  
      model.compile(loss= 'mse', optimizer= 'adam')
      return model
  
  clf = DNN(1, 1, useBN = True)
  clf.fit(x_train, y_train, epochs= 30, batch_size = 100, verbose=2, validation_data = (x_test, y_test))
  
  y_pred = clf.predict(x_test)
  def mse(y_pred, y_test):
      return np.mean(np.square(y_pred - y_test))
  print('final result', mse(y_pred, y_test))
  </code></pre>
  
  <p>The input <code>x</code> is like this shape:</p>
  
  <p><a href="https://i.sstatic.net/cFodJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cFodJ.png" alt="enter image description here"></a></p>
  
  <p>If I add BN layers, the result is shown as follows:</p>
  
  <pre><code>Epoch 27/30
   - 0s - loss: 56.2231 - val_loss: 47.5757
  Epoch 28/30
   - 0s - loss: 55.1271 - val_loss: 60.4838
  Epoch 29/30
   - 0s - loss: 53.9937 - val_loss: 87.3845
  Epoch 30/30
   - 0s - loss: 52.8232 - val_loss: 47.4544
  final result 48.204881459013244
  </code></pre>
  
  <p>If I don't add BN layers, the predicted result is better:</p>
  
  <pre><code>Epoch 27/30
   - 0s - loss: 2.6863 - val_loss: 0.8924
  Epoch 28/30
   - 0s - loss: 2.6562 - val_loss: 0.9120
  Epoch 29/30
   - 0s - loss: 2.6440 - val_loss: 0.9027
  Epoch 30/30
   - 0s - loss: 2.6225 - val_loss: 0.9022
  final result 0.9021717561981543
  </code></pre>
  
  <p>Anyone knows the theory about why BN is not suitable for non-gaussian data ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-03T09:46:20.860Z
Comments.Text: Interesting question, if you do the same thing with gaussian data, does BN work properly?
Comments.UserDisplayName: Djib2011
Tags: <neural-networks><deep-learning><deep-neural-networks><batch-normalization>
Title: Is batch normalization not suitable for non-gaussian input?
UserDisplayName: rosefun
ThreadText: # Is batch normalization not suitable for non-gaussian input?
  
  (Asked by: username_0 on 2019-11-03)
  
  I generate some non-Gaussian data, and use two kinds of DNN models, one with BN and the other without BN.
  
  I find that the model DNN with BN can't predict well.
  
  The codes is shown as follow:
  
  ```
  import numpy as np
  import scipy.stats
  import matplotlib.pyplot as plt
  from keras.models import Sequential
  from keras.layers import Dense,Dropout,Activation, BatchNormalization
  
  np.random.seed(1)
  
  # generate non-gaussian data
  def generate_data():
      distribution = scipy.stats.gengamma(1, 70, loc=10, scale=100)
      x = distribution.rvs(size=10000)
      # plt.hist(x)
      # plt.show()
      print ('[mean, var, skew, kurtosis]', distribution.stats('mvsk'))
  
      y = np.sin(x) + np.cos(x) + np.sqrt(x)
      plt.hist(y)
      # plt.show()
      # print(y)
      return x ,y 
  
  x, y = generate_data()
  
  x_train = x[:int(len(x)*0.8)]
  y_train = y[:int(len(y)*0.8)]
  x_test = x[int(len(x)*0.8):]
  y_test = y[int(len(y)*0.8):]
  
  def DNN(input_dim, output_dim, useBN = True):
      '''
      定义一个DNN model
      '''
      model=Sequential()
  
      model.add(Dense(128,input_dim= input_dim))
      if useBN:
          model.add(BatchNormalization())
      model.add(Activation('tanh'))
      model.add(Dropout(0.5))
  
      model.add(Dense(50))
      if useBN:
          model.add(BatchNormalization())
      model.add(Activation('tanh'))
      model.add(Dropout(0.5))
  
      model.add(Dense(output_dim))
      if useBN:
          model.add(BatchNormalization())
      model.add(Activation('relu'))
  
      model.compile(loss= 'mse', optimizer= 'adam')
      return model
  
  clf = DNN(1, 1, useBN = True)
  clf.fit(x_train, y_train, epochs= 30, batch_size = 100, verbose=2, validation_data = (x_test, y_test))
  
  y_pred = clf.predict(x_test)
  def mse(y_pred, y_test):
      return np.mean(np.square(y_pred - y_test))
  print('final result', mse(y_pred, y_test))
  
  ```
  
  The input `x` is like this shape:
  
  [![enter image description here](https://i.sstatic.net/cFodJ.png)](https://i.sstatic.net/cFodJ.png)
  
  If I add BN layers, the result is shown as follows:
  
  ```
  Epoch 27/30
   - 0s - loss: 56.2231 - val_loss: 47.5757
  Epoch 28/30
   - 0s - loss: 55.1271 - val_loss: 60.4838
  Epoch 29/30
   - 0s - loss: 53.9937 - val_loss: 87.3845
  Epoch 30/30
   - 0s - loss: 52.8232 - val_loss: 47.4544
  final result 48.204881459013244
  
  ```
  
  If I don't add BN layers, the predicted result is better:
  
  ```
  Epoch 27/30
   - 0s - loss: 2.6863 - val_loss: 0.8924
  Epoch 28/30
   - 0s - loss: 2.6562 - val_loss: 0.9120
  Epoch 29/30
   - 0s - loss: 2.6440 - val_loss: 0.9027
  Epoch 30/30
   - 0s - loss: 2.6225 - val_loss: 0.9022
  final result 0.9021717561981543
  
  ```
  
  Anyone knows the theory about why BN is not suitable for non-gaussian data ?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, batch-normalization
  
  > **Comments:**
  > **username_1**: Interesting question, if you do the same thing with gaussian data, does BN work properly?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-03. Score: 1
  
  So batch-normalization helps descent based learning have an easier time traversing the loss manifold, but in your case you use it along with a relu as a final activation is problematic, it means the output is relatively associated with the other samples in the batch.
  
  Remove that last BN and you get better results, but also understand BN is inherently problematic for this task. Think of DNNs as featurizers, and BN in this case takes out the 2 batch-wide statistics which if they don't align to the initial distribution will cause error, which will lead to error in the output. In theory if BN gets the perfect statistics of the gaussian it should not matter too much, so one thing I tried with your code was remove last BN and increase N to 100,000 while increasing the batch size to 10000 and you see a huge boost in performance.
  
  
  ---
  
  