Answers.Body: <p>The purpose of a clockwork RNN is to help with long term dependencies. Let's say in this case, we have a sentence that starts with &quot;John went to...&quot; and at no point again is John's name mentioned throughout the few paragraphs we are passing to our model.</p>
  <p>As mentioned in the paper, the most common method to combat this (at the time) was using an LSTM that stored long term data in it's cell state, or as put in the paper:</p>
  <blockquote>
  <p>[an LSTM] uses a specialized architecture that allows information to be stored in a linear unit called a <em>constant error carousel</em> (CEC) indefinitely</p>
  </blockquote>
  <p>However, this requires a whole heap of extra parameters in order to work (input gate, output gate and forget gate which all require parameters). So proposed was the CW-RNN.</p>
  <p>The fundamental idea behind a clockwork RNN is to have &quot;modules&quot; computed periodically. First to clear things up, a <strong>timestep is 1 input into the model</strong>. So in the case of our example, if we're inputting character by character, timestep 1 is the character &quot;J&quot;, timestep 2 is &quot;h&quot; and so on; &quot;o&quot;, &quot;n&quot;, &quot; &quot;, &quot;w&quot;...</p>
  <p>So what is the <strong>clock-rate</strong>? Well, it's simply how often each module of the CW-RNN is computed. Let's say the hidden layer of the clockwork RNN is split into 8 modules, which I will reference as <span class="math-container">$M_1, M_2, M_3 ... M_8$</span>, and the associated <strong>clock-rates</strong> for each of these modules are the powers of 2, so: <span class="math-container">$1, 2, 4, 8, 16, 32, 64$</span> and <span class="math-container">$128$</span>.</p>
  <p>Before I continue I want to note a potential discrepancy, if you consider the first input to be timestep 0, then it changes how this executed (all modules would be activated at the first timestep), however if you consider it to be 1 (like I do for this example), only module 1 would be executed (again, in this example).</p>
  <p>So we're at timestep 1, ie &quot;J&quot;, so we check the timestep against the <strong>clock-rates</strong> of each module to determine which ones will be executed in the computation of the hidden state and output for this timestep. To do this, we take the mod of the timestep against the <strong>time-rate</strong>, so: <code>1 mod 1, 1 mod 2, 1 mod 4 ... 1 mod 128</code> and if it equals 0 (basically, is the timestep a multiple of the <strong>time-rate</strong>) then that module will be executed at this timestep. So in this case <span class="math-container">$M_1$</span> will be executed. When we input &quot;h&quot; at timestep 2, <span class="math-container">$M_1$</span> and <span class="math-container">$M_2$</span> will both be executed, and will equally contribute to the hidden state (ie, they will be added together) and the output.</p>
  <p>By performing calculations this way, each module can all simultaneously be responsible for information over different time periods, for example <span class="math-container">$M_8$</span> which is only executed every 128 timesteps will be responsible for very long term dependencies, but <span class="math-container">$M_1$</span> will cover short term dependencies.</p>
  <p>So in basic terms, <strong>clock-rate refers to how often (what timestep interval) a given module of a Clockwork RNN is computed</strong></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-16T15:28:43.367Z
Answers.Text: Thanks for the answer. So does a CW-RNN actually produce superior results to the LSTM (for this purpose of 'long term memory')?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T02:41:46.327Z
Answers.Text: According to the paper, yes it does "CW-RNNs beat both RNN and LSTM networks by a considerable margin of 8-20% on average" for classification and " CW-RNNs give by far the best results, with the
  smallest one being roughly on par with the second-biggest
  LSTM network" for sequence generation
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-13T06:09:01.603Z
Answers.LastActivityDate: 2021-04-13T06:09:01.603Z
Answers.UserDisplayName: Recessive
Answers.Body: <p>It seems this paper defines a &quot;clock period <span class="math-container">${T}_n$</span>&quot; that it uses to express the topology of the network: &quot;Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period <span class="math-container">$T_i$</span> is smaller than period <span class="math-container">$T_j$</span>.&quot;.</p>
  <p>This definition is, however, only in this paper, as far as I know. It wouldn't make sense to have a &quot;clock period&quot; (or &quot;clock rate&quot;) on other, more self-similar, RNNs. However &quot;time t&quot; is something that is defined to express the amount of times the RNN has recursed, &quot;t times&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-10T19:46:08.683Z
Answers.Text: Thanks for the answer. So what exactly does "clock rate" mean in the context of the linked paper?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-10T23:09:34.450Z
Answers.Text: @ThePointer "clock rate" refers on how modules process data. The idea of the paper is to compute information at different "rates", for this, it uses modules, the modules are connected to others according to the module's clock period.  This creates a network where information is processed by each module at different rates. Quote: "Each module $i$ is evaluated only every $T_i$-th time step", this would be a 'rate' of computation of incoming data.
Answers.UserDisplayName: EmmanuelMess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-10T06:18:46.170Z
Answers.LastActivityDate: 2021-04-10T06:18:46.170Z
Answers.UserDisplayName: EmmanuelMess
Body: <p>I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see <a href="https://arxiv.org/abs/1402.3511" rel="nofollow noreferrer">this</a> paper. However, I cannot find any explanations for what this means. What does 'clock rate' mean in this context?</p>
  
Tags: <recurrent-neural-networks>
Title: What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?
UserDisplayName: The Pointer
ThreadText: # What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?
  
  (Asked by: username_0 on 2021-04-07)
  
  I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see [this](https://arxiv.org/abs/1402.3511) paper. However, I cannot find any explanations for what this means. What does 'clock rate' mean in this context?
  
  **Tags:** recurrent-neural-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-04-13. Score: 1 | Accepted answer
  
  The purpose of a clockwork RNN is to help with long term dependencies. Let's say in this case, we have a sentence that starts with "John went to..." and at no point again is John's name mentioned throughout the few paragraphs we are passing to our model.
  
  As mentioned in the paper, the most common method to combat this (at the time) was using an LSTM that stored long term data in it's cell state, or as put in the paper:
  
  > [an LSTM] uses a specialized architecture that allows information to be stored in a linear unit called a *constant error carousel* (CEC) indefinitely
  
  However, this requires a whole heap of extra parameters in order to work (input gate, output gate and forget gate which all require parameters). So proposed was the CW-RNN.
  
  The fundamental idea behind a clockwork RNN is to have "modules" computed periodically. First to clear things up, a **timestep is 1 input into the model**. So in the case of our example, if we're inputting character by character, timestep 1 is the character "J", timestep 2 is "h" and so on; "o", "n", " ", "w"...
  
  So what is the **clock-rate**? Well, it's simply how often each module of the CW-RNN is computed. Let's say the hidden layer of the clockwork RNN is split into 8 modules, which I will reference as $M\_1, M\_2, M\_3 ... M\_8$, and the associated **clock-rates** for each of these modules are the powers of 2, so: $1, 2, 4, 8, 16, 32, 64$ and $128$.
  
  Before I continue I want to note a potential discrepancy, if you consider the first input to be timestep 0, then it changes how this executed (all modules would be activated at the first timestep), however if you consider it to be 1 (like I do for this example), only module 1 would be executed (again, in this example).
  
  So we're at timestep 1, ie "J", so we check the timestep against the **clock-rates** of each module to determine which ones will be executed in the computation of the hidden state and output for this timestep. To do this, we take the mod of the timestep against the **time-rate**, so: `1 mod 1, 1 mod 2, 1 mod 4 ... 1 mod 128` and if it equals 0 (basically, is the timestep a multiple of the **time-rate**) then that module will be executed at this timestep. So in this case $M\_1$ will be executed. When we input "h" at timestep 2, $M\_1$ and $M\_2$ will both be executed, and will equally contribute to the hidden state (ie, they will be added together) and the output.
  
  By performing calculations this way, each module can all simultaneously be responsible for information over different time periods, for example $M\_8$ which is only executed every 128 timesteps will be responsible for very long term dependencies, but $M\_1$ will cover short term dependencies.
  
  So in basic terms, **clock-rate refers to how often (what timestep interval) a given module of a Clockwork RNN is computed**
  
  > **Comments:**
  > **username_0**: Thanks for the answer. So does a CW-RNN actually produce superior results to the LSTM (for this purpose of 'long term memory')?  
  > **username_1**: According to the paper, yes it does "CW-RNNs beat both RNN and LSTM networks by a considerable margin of 8-20% on average" for classification and " CW-RNNs give by far the best results, with the smallest one being roughly on par with the second-biggest LSTM network" for sequence generation  
  
  ---
  
  ### Answer by username_2 on 2021-04-10. Score: 1
  
  It seems this paper defines a "clock period ${T}\_n$" that it uses to express the topology of the network: "Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period $T\_i$ is smaller than period $T\_j$.".
  
  This definition is, however, only in this paper, as far as I know. It wouldn't make sense to have a "clock period" (or "clock rate") on other, more self-similar, RNNs. However "time t" is something that is defined to express the amount of times the RNN has recursed, "t times".
  
  > **Comments:**
  > **username_0**: Thanks for the answer. So what exactly does "clock rate" mean in the context of the linked paper?  
  > **username_2**: @username_0 "clock rate" refers on how modules process data. The idea of the paper is to compute information at different "rates", for this, it uses modules, the modules are connected to others according to the module's clock period.  This creates a network where information is processed by each module at different rates. Quote: "Each module $i$ is evaluated only every $T_i$-th time step", this would be a 'rate' of computation of incoming data.  
  
  ---
  
  