Body: <p>I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.</p>
  <p>I was told that batch normalization layers might be trying to find weights that are good for all while training, which is a problem since the distribution of my synthetic data is not exactly equal to the distribution of the real data. So, the idea would be to have different 'copies' of the weights of batch normalization layers. So that the neural network estimates different weights for synthetic and real data, and uses just the weights of real data for evaluation.</p>
  <p>My question is, how to perform batch normalization in the aforementioned case? Is there already an implementation of batch norm layers in PyTorch that solves the problem?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T12:50:57.207Z
Comments.Text: Does the second part of the question falls in to the category of **asking for an API, library, or dataset (to solve a specific problem).** which is off-topic? @nbro
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:00:08.180Z
Comments.Text: I am interested in first part of the question.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:00:24.457Z
Comments.Text: Oh, in this case? Where should i post this question?
Comments.UserDisplayName: Manveru
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:01:26.963Z
Comments.Text: The question is nice. But i am not sure about the part  **Is there already an implementation of batch norm layers in PyTorch that solves the problem?** only
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:02:54.083Z
Comments.Text: The question **what is the best and easiest way to implement that?** is on-topic since it is asking for reconciling batch norm over multiple distributions.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:03:02.400Z
Comments.Text: Oh I see. I would like to implement it, but I dont know enough pytorch to change it alone. That is why I would like some hints
Comments.UserDisplayName: Manveru
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:07:28.887Z
Comments.Text: You can remove the words **best** and **easiest** since they are superlative. Instead you can ask, **how to perform batch normalization in aforementioned case?**
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T13:18:00.743Z
Comments.Text: @hanugm If "Is there already an implementation of batch norm layers in PyTorch that solves the problem?" is just a secondary/additional question, i.e. it's not the main question, and by "how to perform batch normalization in the aforementioned case?" the OP is asking, for example, for the pseudocode that would allow them to implement the approach to solve their problem, the question is fine (I think).
Comments.UserDisplayName: nbro
Tags: <deep-learning><datasets><pytorch><batch-normalization><data-augmentation>
Title: Batch normalization for multiple datasets?
UserDisplayName: Manveru
ThreadText: # Batch normalization for multiple datasets?
  
  (Asked by: username_0 on 2021-07-26)
  
  I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
  
  I was told that batch normalization layers might be trying to find weights that are good for all while training, which is a problem since the distribution of my synthetic data is not exactly equal to the distribution of the real data. So, the idea would be to have different 'copies' of the weights of batch normalization layers. So that the neural network estimates different weights for synthetic and real data, and uses just the weights of real data for evaluation.
  
  My question is, how to perform batch normalization in the aforementioned case? Is there already an implementation of batch norm layers in PyTorch that solves the problem?
  
  **Tags:** deep-learning, datasets, pytorch, batch-normalization, data-augmentation
  
  > **Comments:**
  > **username_1**: Does the second part of the question falls in to the category of **asking for an API, library, or dataset (to solve a specific problem).** which is off-topic? @username_2  
  > **username_1**: I am interested in first part of the question.  
  > **username_0**: Oh, in this case? Where should i post this question?  
  > **username_1**: The question is nice. But i am not sure about the part  **Is there already an implementation of batch norm layers in PyTorch that solves the problem?** only  
  > **username_1**: The question **what is the best and easiest way to implement that?** is on-topic since it is asking for reconciling batch norm over multiple distributions.  
  > **username_0**: Oh I see. I would like to implement it, but I dont know enough pytorch to change it alone. That is why I would like some hints  
  > **username_1**: You can remove the words **best** and **easiest** since they are superlative. Instead you can ask, **how to perform batch normalization in aforementioned case?**  
  > **username_2**: @username_1 If "Is there already an implementation of batch norm layers in PyTorch that solves the problem?" is just a secondary/additional question, i.e. it's not the main question, and by "how to perform batch normalization in the aforementioned case?" the OP is asking, for example, for the pseudocode that would allow them to implement the approach to solve their problem, the question is fine (I think).  
  
  ---
  
  ## 0 answers
  
  