Body: <p>I have a time-varying input size vector for a RNN. However, I am facing some difficulties understanding how to deal with my network weights when the input changes. </p>
  
  <p>Say we have a set of natural positive integers
  <span class="math-container">$$
  \Gamma=\{1,2,\dots,F\},
  $$</span>
  where <span class="math-container">$F=100$</span> for the sake of the example.</p>
  
  <p>A valid observation vector of my agent at time <span class="math-container">$t$</span> might be 
  <span class="math-container">$$
  \gamma_t=[1,3,5,1].
  $$</span>
  Thus, at time <span class="math-container">$t$</span> a set of weights will be produced by my RNN, according to <span class="math-container">$\gamma_t$</span>. Sat that at time <span class="math-container">$t+1$</span>, my observation vector changes as
  <span class="math-container">$$
  \gamma_{t+1}=[3,5,8],
  $$</span>
  and there is my problem. If I now continue training my RNN with the previous weights, the output would be inevitably affected. Also, Which weight shall I remove? I see RNN can face the issue but how shall I deal with the previously computed weights? Which one to remove? How to initialize a new one in case the cardinality of <span class="math-container">$\gamma_{t+1}$</span> is higher than that of <span class="math-container">$\gamma_t$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-11T12:18:35.053Z
Comments.Text: is each $\gamma_t$ a new input to the RNN? or is the temporal component that it feeds in all $\gamma$'s at once? Also what weight are you referring to, when asking about "removing"?
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-11T14:51:14.170Z
Comments.Text: $\gamma_t$ is the input of the RNN at time $t$. Let's say, $\gamma_t$ is a vector of requests that have to be satisfied by the agent. It might be that a request (num. $1$ in my example) is satisfied earlier and so the vector $\gamma_t$ will not present that element at time $t+1$. At the same time, a new request comes in (num. $8$ in my example) and so $\gamma_{t+1}$ will present this new value.
Comments.UserDisplayName: EmG
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-11T14:51:18.120Z
Comments.Text: About the weights, I'm wondering if this change on the input layer would create some confusion to the RNN which will have a set of weights for the linear combination that refer to the previous condition. Have I been clear for what I intend as 'weights'? Thanks for your answer
Comments.UserDisplayName: EmG
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-11T15:03:08.063Z
Comments.Text: i see, so youre problem is like one long temporal vector, but for the rnn you only let it see parts that fall into some form of bin (where bins overlap?). Can you clarify your problem a bit more specifically?
Comments.UserDisplayName: mshlis
Tags: <neural-networks><recurrent-neural-networks>
Title: RNN weights when varying the input size
UserDisplayName: EmG
ThreadText: # RNN weights when varying the input size
  
  (Asked by: username_0 on 2019-06-10)
  
  I have a time-varying input size vector for a RNN. However, I am facing some difficulties understanding how to deal with my network weights when the input changes.
  
  Say we have a set of natural positive integers
  $$
  \Gamma=\{1,2,\dots,F\},
  $$
  where $F=100$ for the sake of the example.
  
  A valid observation vector of my agent at time $t$ might be
  $$
  \gamma\_t=[1,3,5,1].
  $$
  Thus, at time $t$ a set of weights will be produced by my RNN, according to $\gamma\_t$. Sat that at time $t+1$, my observation vector changes as
  $$
  \gamma\_{t+1}=[3,5,8],
  $$
  and there is my problem. If I now continue training my RNN with the previous weights, the output would be inevitably affected. Also, Which weight shall I remove? I see RNN can face the issue but how shall I deal with the previously computed weights? Which one to remove? How to initialize a new one in case the cardinality of $\gamma\_{t+1}$ is higher than that of $\gamma\_t$?
  
  **Tags:** neural-networks, recurrent-neural-networks
  > **Comments:**
  > **username_1**: is each $\gamma_t$ a new input to the RNN? or is the temporal component that it feeds in all $\gamma$'s at once? Also what weight are you referring to, when asking about "removing"?  
  > **username_0**: $\gamma_t$ is the input of the RNN at time $t$. Let's say, $\gamma_t$ is a vector of requests that have to be satisfied by the agent. It might be that a request (num. $1$ in my example) is satisfied earlier and so the vector $\gamma_t$ will not present that element at time $t+1$. At the same time, a new request comes in (num. $8$ in my example) and so $\gamma_{t+1}$ will present this new value.  
  > **username_0**: About the weights, I'm wondering if this change on the input layer would create some confusion to the RNN which will have a set of weights for the linear combination that refer to the previous condition. Have I been clear for what I intend as 'weights'? Thanks for your answer  
  > **username_1**: i see, so youre problem is like one long temporal vector, but for the rnn you only let it see parts that fall into some form of bin (where bins overlap?). Can you clarify your problem a bit more specifically?  
  
  ---
  
  ## 0 answers
  
  