Answers.Body: <p>The strategy is turning b2 into a separate model, initialise b2 the way it should be, and train your network without b2 as usual.</p>
  <p>In the middle of the main network, combine the output of b1 layer and b2 network using concatenation function, for example in TensorFlow:</p>
  <pre><code># Axis 0 is the batch dimension, axis 1 for the dimension of value in every sample
  tf.concat([b1out,b2out],axis=1)
  </code></pre>
  <p>Example source code (paste to Google Colab to test):</p>
  <pre><code>%tensorflow_version 2.x
  %reset -f
  
  import tensorflow                   as     tf
  from   tensorflow.keras             import *
  from   tensorflow.keras.layers      import *
  from   tensorflow.keras.activations import *
  from   tensorflow.keras.models      import *
  from   tensorflow.keras.callbacks   import *
  
  class MyModel(Model):
      def __init__(self):
          super(MyModel,self).__init__()
          self.dense1 = Dense(200, activation=relu)
          self.dense2 = Dense(1,   activation=tf.identity)
  
      @tf.function
      def call(self,x):
          # MIND THE CONCATENATION IN THIS PART:
          h1a = self.dense1(x)
          h1b = b2(x) # b2 won't get trained, so it stays fixed
  
          # CONCATENATION AT AXIS 1, COZ AXIS 0 IS BATCH DIMENSION
          h1  = tf.concat([h1a,h1b],axis=1) 
          u   = self.dense2(h1)
          return u
  
  class ModelB2(Model):
      def __init__(self):
          super(ModelB2,self).__init__()
          self.dense1 = Dense(200)
          # self.dense2 = ...
          # Init weights of B here or B is pre-trained model
  
      @tf.function
      def call(self,x):
          u = self.dense1(x)
          return u
  
  # PROGRAMME ENTRY POINT ========================================================
  if __name__==&quot;__main__&quot;:
      inp = [[1,2,3,4,5,6,7],[7,6,5,4,3,2,1]] # Example values
      exp = [[0],            [1]            ]
  
      mm = MyModel()
      b2 = ModelB2()
  
      mm.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
      b2.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
      mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
      b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
  
      print(&quot;\nbefore training:&quot;)
      print(&quot;mm weights:&quot;,mm.get_weights()[0][0][:3],&quot;...&quot;)
      print(&quot;b2 weights:&quot;,b2.get_weights()[0][0][:3],&quot;...&quot;)
      print(&quot;mm loss:&quot;,mm_loss)
      print(&quot;b2 loss:&quot;,b2_loss)
  
      mm.fit(x=inp,y=exp, batch_size=len(inp), epochs=500, verbose=0)
      mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) 
      b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) 
  
      print(&quot;\nafter training:&quot;)
      print(&quot;mm weights:&quot;,mm.get_weights()[0][0][:3],&quot;...&quot;)
      print(&quot;b2 weights:&quot;,b2.get_weights()[0][0][:3],&quot;... &lt;-- UNCHANGED AS WANTED&quot;)
      print(&quot;mm loss:&quot;,mm_loss)
      print(&quot;b2 loss:&quot;,b2_loss,&quot;&lt;-- UNCHANGED AS WANTED&quot;)
  # EOF
  </code></pre>
  <p>Google colab: <a href="https://colab.research.google.com" rel="nofollow noreferrer">https://colab.research.google.com</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-05T04:15:29.403Z
Answers.LastActivityDate: 2021-12-21T04:52:37.960Z
Answers.LastEditDate: 2021-12-21T04:52:37.960Z
Answers.UserDisplayName: Dan D.
Body: <p>I am a beginner in neural networks. I am building a neural network with 3 layers. The input <span class="math-container">$X$</span> has 7 features and the output <span class="math-container">$Y$</span> is a real number. In the hidden layer, there are two nodes. The bottom node contains weights and biases which should be hard set.</p>
  <p><a href="https://i.sstatic.net/CshpZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CshpZ.png" alt="Sample neural network trying to train where the red weights and bias term b2 should be hard set." /></a></p>
  <p>Now, I want to train this neural network with the training data <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>, such that the red weights are held constant while all other weights are learnable.</p>
  <p>Is there a way of doing this during the training of the neural network? I'm using TensorFlow and Keras, so, if you could provide also the code necessary to do this, that would be very useful.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-05T02:31:16.347Z
Comments.Text: maybe set weights for b2 or train it first, and use it as a function
Comments.UserDisplayName: Dan D.
Tags: <neural-networks><tensorflow><keras><weights>
Title: How to train a neural network with few weights and biases held constant?
UserDisplayName: xyztg
ThreadText: # How to train a neural network with few weights and biases held constant?
  
  (Asked by: username_0 on 2021-03-04)
  
  I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom node contains weights and biases which should be hard set.
  
  [![Sample neural network trying to train where the red weights and bias term b2 should be hard set.](https://i.sstatic.net/CshpZ.png)](https://i.sstatic.net/CshpZ.png)
  
  Now, I want to train this neural network with the training data $X$ and $Y$, such that the red weights are held constant while all other weights are learnable.
  
  Is there a way of doing this during the training of the neural network? I'm using TensorFlow and Keras, so, if you could provide also the code necessary to do this, that would be very useful.
  
  **Tags:** neural-networks, tensorflow, keras, weights
  
  > **Comments:**
  > **username_1**: maybe set weights for b2 or train it first, and use it as a function  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-05. Score: 1
  
  The strategy is turning b2 into a separate model, initialise b2 the way it should be, and train your network without b2 as usual.
  
  In the middle of the main network, combine the output of b1 layer and b2 network using concatenation function, for example in TensorFlow:
  
  ```
  # Axis 0 is the batch dimension, axis 1 for the dimension of value in every sample
  tf.concat([b1out,b2out],axis=1)
  
  ```
  
  Example source code (paste to Google Colab to test):
  
  ```
  %tensorflow_version 2.x
  %reset -f
  
  import tensorflow                   as     tf
  from   tensorflow.keras             import *
  from   tensorflow.keras.layers      import *
  from   tensorflow.keras.activations import *
  from   tensorflow.keras.models      import *
  from   tensorflow.keras.callbacks   import *
  
  class MyModel(Model):
      def __init__(self):
          super(MyModel,self).__init__()
          self.dense1 = Dense(200, activation=relu)
          self.dense2 = Dense(1,   activation=tf.identity)
  
      @tf.function
      def call(self,x):
          # MIND THE CONCATENATION IN THIS PART:
          h1a = self.dense1(x)
          h1b = b2(x) # b2 won't get trained, so it stays fixed
  
          # CONCATENATION AT AXIS 1, COZ AXIS 0 IS BATCH DIMENSION
          h1  = tf.concat([h1a,h1b],axis=1) 
          u   = self.dense2(h1)
          return u
  
  class ModelB2(Model):
      def __init__(self):
          super(ModelB2,self).__init__()
          self.dense1 = Dense(200)
          # self.dense2 = ...
          # Init weights of B here or B is pre-trained model
  
      @tf.function
      def call(self,x):
          u = self.dense1(x)
          return u
  
  # PROGRAMME ENTRY POINT ========================================================
  if __name__=="__main__":
      inp = [[1,2,3,4,5,6,7],[7,6,5,4,3,2,1]] # Example values
      exp = [[0],            [1]            ]
  
      mm = MyModel()
      b2 = ModelB2()
  
      mm.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
      b2.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
      mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
      b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
  
      print("\nbefore training:")
      print("mm weights:",mm.get_weights()[0][0][:3],"...")
      print("b2 weights:",b2.get_weights()[0][0][:3],"...")
      print("mm loss:",mm_loss)
      print("b2 loss:",b2_loss)
  
      mm.fit(x=inp,y=exp, batch_size=len(inp), epochs=500, verbose=0)
      mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) 
      b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) 
  
      print("\nafter training:")
      print("mm weights:",mm.get_weights()[0][0][:3],"...")
      print("b2 weights:",b2.get_weights()[0][0][:3],"... <-- UNCHANGED AS WANTED")
      print("mm loss:",mm_loss)
      print("b2 loss:",b2_loss,"<-- UNCHANGED AS WANTED")
  # EOF
  
  ```
  
  Google colab: <https://colab.research.google.com>
  
  
  ---
  
  