Answers.Body: <p>Yes that is indeed the case. You must store the used normalization weights, and use them also with the new data. Otherwise their distribution would be different, and your model's performance will be very poor.</p>
  <p>Actually I got a crazy(?) idea just now, could you add a BatchNormalization layer right after the network's input layer? This way it would be automatically be stored as part of the model. I haven't ever tested this idea, or seen it being used anywhere.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T12:04:11.973Z
Answers.Text: Your idea is already here: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Normalization :) This performs standardization, so if one requires a different normalization (like min-max scaling) is possible to create a custom pre-processing layer with weights that would be learned before training, then fixed, and finally saved along the model's weights
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-19T08:16:25.667Z
Answers.Text: Ahaa I didn't know about that one, it seems like a better fit for this purpose rather than the BatchNormalization.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-20T18:52:51.047Z
Answers.Text: Thank you both very much for these helpful thoughts and alternative approaches.
Answers.UserDisplayName: Pittsburgh DBA
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T10:38:41.003Z
Answers.LastActivityDate: 2023-09-18T10:38:41.003Z
Answers.UserDisplayName: NikoNyrh
Answers.Body: <p>Given the answer provided by @NikoNyrh is correct, I just want to add how you can make a custom pre-processing layer with tensorflow that you can integrate in your model.</p>
  <p>Assume you want to build a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html" rel="nofollow noreferrer">min-max scaler</a>:</p>
  <pre><code>import tensorflow as tf
  
  from tensorflow.keras.layers.experimental.preprocessing import PreprocessingLayer
  from tensorflow.keras.layers import Input
  
  class MinMaxScaler(PreprocessingLayer):
  
      def __init__(self, **kwargs):
          super().__init__(**kwargs)
  
          # assume normalization should occur on last axis
          self.axis = 1
  
          # weights: to be set in build()
          self.min_value: tf.Variable = None
          self.max_value: tf.Variable = None
  
      def build(self, input_shape):
          super().build(input_shape)
  
          input_shape = tf.TensorShape(input_shape).as_list()
          shape = (input_shape[self.axis],)
  
          self.min_value = self.add_weight(
              name=&quot;min_value&quot;, shape=shape, dtype=self.compute_dtype,
              initializer='zeros', trainable=False)
  
          self.max_value = self.add_weight(
              name=&quot;max_value&quot;, shape=shape, dtype=self.compute_dtype,
              initializer='zeros', trainable=False)
  
      def call(self, inputs):
          # Here the normalization occurs on the model' inputs
          inputs = tf.convert_to_tensor(inputs, dtype=self.compute_dtype)
          _range = self.max_value - self.min_value + tf.keras.backend.epsilon()
          return (inputs - self.min_value) / _range
  
      def update_state(self, data):
          # Normalization logic here
          data = tf.convert_to_tensor(data, dtype=self.compute_dtype)
  
          # find min-max boundaries
          batch_min = tf.reduce_min(data, axis=0)
          batch_max = tf.reduce_max(data, axis=0)
  
          # update the weights
          self.min_value.assign(tf.minimum(self.min_value, batch_min))
          self.max_value.assign(tf.maximum(self.max_value, batch_max))
  
      def reset_state(self):
          self.min_value.assign(tf.zeros_like(self.min_value))
          self.max_value.assign(tf.zeros_like(self.max_value))
  </code></pre>
  <p>You can use the <code>MinMaxScaler</code> layer within a keras model easily:</p>
  <pre><code># build some model
  inp = Input(input_shape=your_input_shape)
  scaler = MinMaxScaler()
  norm_inp = scaler(inp)
  # ... hidden layers
  out = ...
  
  model = tf.keras.Model(inp, out)
  model.compile(...)
  
  # before training the model, you should call adapt() to find the normalization weights
  scaler.adapt(some_data)
  print(scaler.min_value, scaler.max_value)
  
  # finally you can train the model
  model.fit(...)
  
  # save everything
  model.save_weights()  # for example
  </code></pre>
  <p>Basically, the <code>adapt()</code> method computes the weights/parameters that are used to normalize the data when calling <code>model(x)</code>, for example during inference. The custom layer can be included in a <code>tf.keras.Model</code> having all the saving and loading benefits, i.e., you don't have to keep track of the normalization statistics by yourself. Last advantage, is that the normalization/scaling logic can run on CPU/GPU and TPU as well.</p>
  <p>Indeed, this is a minimum working example so in real code you want to add more functionalities and checks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-20T18:58:44.150Z
Answers.Text: This is terrific. Thank you for working up this helpful example. Now this makes it difficult to determine which answer to accept!
Answers.UserDisplayName: Pittsburgh DBA
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-21T13:27:27.467Z
Answers.Text: @PittsburghDBA Thanks. Well, you can upvote both and accept what you find the most useful :)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-20T17:29:25.367Z
Answers.LastActivityDate: 2023-09-20T17:29:25.367Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I trained a Keras Network. During training, I would first initialize a normalizer from the values in the entire dataset, then partition into train, test and validation datasets. After partitioning, I applied the normalizer to each set.</p>
  <p>Now I wish to use the network in production.</p>
  <p>As the new data arrive, and I wish to execute the network, I obviously need to normalize the input. Must I pre-initialize the normalizer with the values from the original training set, in order for the normalization step to be valid for the network as trained?</p>
  <p>I can't envision how it would work otherwise. Thank you.</p>
  
Tags: <neural-networks><keras><normalisation>
Title: Must I "prime" my normalizer with the same data I trained it with in order to use it?
UserDisplayName: Pittsburgh DBA
ThreadText: # Must I "prime" my normalizer with the same data I trained it with in order to use it?
  
  (Asked by: username_0 on 2023-09-17)
  
  I trained a Keras Network. During training, I would first initialize a normalizer from the values in the entire dataset, then partition into train, test and validation datasets. After partitioning, I applied the normalizer to each set.
  
  Now I wish to use the network in production.
  
  As the new data arrive, and I wish to execute the network, I obviously need to normalize the input. Must I pre-initialize the normalizer with the values from the original training set, in order for the normalization step to be valid for the network as trained?
  
  I can't envision how it would work otherwise. Thank you.
  
  **Tags:** neural-networks, keras, normalisation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-09-18. Score: 2
  
  Yes that is indeed the case. You must store the used normalization weights, and use them also with the new data. Otherwise their distribution would be different, and your model's performance will be very poor.
  
  Actually I got a crazy(?) idea just now, could you add a BatchNormalization layer right after the network's input layer? This way it would be automatically be stored as part of the model. I haven't ever tested this idea, or seen it being used anywhere.
  
  > **Comments:**
  > **username_2**: Your idea is already here: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Normalization :) This performs standardization, so if one requires a different normalization (like min-max scaling) is possible to create a custom pre-processing layer with weights that would be learned before training, then fixed, and finally saved along the model's weights  
  > **username_1**: Ahaa I didn't know about that one, it seems like a better fit for this purpose rather than the BatchNormalization.  
  > **username_0**: Thank you both very much for these helpful thoughts and alternative approaches.  
  
  ---
  
  ### Answer by username_2 on 2023-09-20. Score: 1
  
  Given the answer provided by @NikoNyrh is correct, I just want to add how you can make a custom pre-processing layer with tensorflow that you can integrate in your model.
  Assume you want to build a min-max scaler:
  import tensorflow as tf
  
  from tensorflow.keras.layers.experimental.preprocessing import PreprocessingLayer
  from tensorflow.keras.layers import Input
  
  class MinMaxScaler(PreprocessingLayer):
  
      def __init__(self, **kwargs):
          super().__init__(**kwargs)
  
          # assume normalization should occur on last axis
          self.axis = 1
  
          # weights: to be set in build()
          self.min_value: tf.Variable = None
          self.max_value: tf.Variable = None
  
      def build(self, input_shape):
          super().build(input_shape)
  
          input_shape = tf.TensorShape(input_shape).as_list()
          shape = (input_shape[self.axis],)
  
          self.min_value = self.add_weight(
              name="min_value", shape=shape, dtype=self.compute_dtype,
              initializer='zeros', trainable=False)
  
          self.max_value = self.add_weight(
              name="max_value", shape=shape, dtype=self.compute_dtype,
              initializer='zeros', trainable=False)
  
      def call(self, inputs):
          # Here the normalization occurs on the model' inputs
          inputs = tf.convert_to_tensor(inputs, dtype=self.compute_dtype)
          _range = self.max_value - self.min_value + tf.keras.backend.epsilon()
          return (inputs - self.min_value) / _range
  
      def update_state(self, data):
          # Normalization logic here
          data = tf.convert_to_tensor(data, dtype=self.compute_dtype)
  
          # find min-max boundaries
          batch_min = tf.reduce_min(data, axis=0)
          batch_max = tf.reduce_max(data, axis=0)
  
          # update the weights
          self.min_value.assign(tf.minimum(self.min_value, batch_min))
          self.max_value.assign(tf.maximum(self.max_value, batch_max))
  
      def reset_state(self):
          self.min_value.assign(tf.zeros_like(self.min_value))
          self.max_value.assign(tf.zeros_like(self.max_value))
  
  You can use the MinMaxScaler layer within a keras model easily:
  # build some model
  inp = Input(input_shape=your_input_shape)
  scaler = MinMaxScaler()
  norm_inp = scaler(inp)
  # ... hidden layers
  out = ...
  
  model = tf.keras.Model(inp, out)
  model.compile(...)
  
  # before training the model, you should call adapt() to find the normalization weights
  scaler.adapt(some_data)
  print(scaler.min_value, scaler.max_value)
  
  # finally you can train the model
  model.fit(...)
  
  # save everything
  model.save_weights()  # for example
  
  Basically, the adapt() method computes the weights/parameters that are used to normalize the data when calling model(x), for example during inference. The custom layer can be included in a tf.keras.Model having all the saving and loading benefits, i.e., you don't have to keep track of the normalization statistics by yourself. Last advantage, is that the normalization/scaling logic can run on CPU/GPU and TPU as well.
  Indeed, this is a minimum working example so in real code you want to add more functionalities and checks.
  
  > **Comments:**
  > **username_0**: This is terrific. Thank you for working up this helpful example. Now this makes it difficult to determine which answer to accept!  
  > **username_2**: @PittsburghDBA Thanks. Well, you can upvote both and accept what you find the most useful :)  
  
  ---
  
  