Answers.Body: <p><strong>Can I initialize the initial state of my RNN to be non-zero?</strong>  Yes, the initial state / weights of a neural network can be initialized to non-zero values.  In fact, the <code>trainable weights</code> can be initialized any way you want.</p>
  <p>Exactly how you do this depends on the deep learning package that you are using (if you are using a package).  For example, if you are using Keras/TensorFlow, there is built-in functionality to <a href="https://keras.io/api/layers/initializers/" rel="nofollow noreferrer">initialize layer weights</a> and <a href="https://keras.io/api/layers/base_layer/#setweights-method" rel="nofollow noreferrer">set model weights</a>.  Here is a nice article demonstrating how this can be done:  <a href="https://androidkt.com/set-custom-weights-keras-using-numpy-array/" rel="nofollow noreferrer">&quot;How to set custom weights in keras using NumPy array&quot;</a>.  The article uses random normal initialization, but you can use your desired initial state / weights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T13:20:25.267Z
Answers.Text: I think OP was maybe asking about the initial state, i.e. first hidden state not the initialization of the network. - @wrek?
  In that case, here is a related thread that might yield an answer: https://ai.stackexchange.com/a/35465/52211
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T14:47:48.453Z
Answers.Text: yes, the $h_0$, the initial hidden state.
Answers.UserDisplayName: wrek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-23T22:15:28.540Z
Answers.LastActivityDate: 2022-10-24T11:51:08.833Z
Answers.LastEditDate: 2022-10-24T11:51:08.833Z
Answers.UserDisplayName: Snehal Patel
Body: <p>Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.</p>
  <p><a href="https://i.sstatic.net/Cpk1Q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Cpk1Q.png" alt="enter image description here" /></a></p>
  
Tags: <recurrent-neural-networks><weights-initialization>
Title: Initial State of RNN
UserDisplayName: wrek
ThreadText: # Initial State of RNN
  
  (Asked by: username_0 on 2022-10-23)
  
  Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.
  
  [![enter image description here](https://i.sstatic.net/Cpk1Q.png)](https://i.sstatic.net/Cpk1Q.png)
  
  **Tags:** recurrent-neural-networks, weights-initialization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-23. Score: 2
  
  **Can I initialize the initial state of my RNN to be non-zero?** Yes, the initial state / weights of a neural network can be initialized to non-zero values. In fact, the `trainable weights` can be initialized any way you want.
  
  Exactly how you do this depends on the deep learning package that you are using (if you are using a package). For example, if you are using Keras/TensorFlow, there is built-in functionality to [initialize layer weights](https://keras.io/api/layers/initializers/) and [set model weights](https://keras.io/api/layers/base_layer/#setweights-method). Here is a nice article demonstrating how this can be done: ["How to set custom weights in keras using NumPy array"](https://androidkt.com/set-custom-weights-keras-using-numpy-array/). The article uses random normal initialization, but you can use your desired initial state / weights.
  
  > **Comments:**
  > **username_2**: I think OP was maybe asking about the initial state, i.e. first hidden state not the initialization of the network. - @username_0? In that case, here is a related thread that might yield an answer: https://ai.stackexchange.com/a/35465/52211  
  > **username_0**: yes, the $h_0$, the initial hidden state.  
  
  ---
  
  