Body: <p>Say I have an application where the frequency of the input is known but can vary widely across sequences. For example, they may be audio recordings acquired at different frequency, or videos that come from surveillance cameras whose framerate can vary from 24fps down to as low as 1fps.</p>
  
  <p>The straightworard thing to do would be to either</p>
  
  <ul>
  <li>resample inputs to a constant frequency</li>
  <li>ignore input frequency and hope the RNN will figure it all out</li>
  </ul>
  
  <p>None sound very appealing. Is there a better way to handle variable input frequency in RNNs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-23T08:30:54.850Z
Comments.Text: Could you be more specific about the nature of the application, because I don't think there is a general answer here? For instance, You cannot realistically expect to perform certain tasks on video at 1fps regardless of the analysis technology - e.g. try to estimate speed of object moving through the view when only one frame captures that it was there in the first place.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-23T08:50:06.660Z
Comments.Text: @NeilSlater For audio, it could be a speech-to-text task. For video, you could imagine to offer an action detection software solution that would not enforce an fps to the client. Obviously you would expect that a lower framerate degrades the precision of the predictions -- otherwise you might just as well resample all records to 1fps.
Comments.UserDisplayName: user209974
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-23T10:13:05.177Z
Comments.Text: There is "variation" and then there is "completely different problem".I cannot give a full answer here (not enough experience with RNNs), but would actually expect a difference of over an order of magnitude in sampling rates is going to have such a deleterious effect that you will want more than one solution, and not attempt to build a single application. Smaller changes, or dealing with an over-abundance of data (e.g. 120fps in 4K resolution from a broadcast-quality camera, where you would downsample just to reduce processing costs) are probably OK.
Comments.UserDisplayName: Neil Slater
Tags: <recurrent-neural-networks>
Title: How to adapt RNNs to variable frequency / framerate of inputs?
UserDisplayName: user209974
ThreadText: # How to adapt RNNs to variable frequency / framerate of inputs?
  
  (Asked by: username_0 on 2018-06-23)
  
  Say I have an application where the frequency of the input is known but can vary widely across sequences. For example, they may be audio recordings acquired at different frequency, or videos that come from surveillance cameras whose framerate can vary from 24fps down to as low as 1fps.
  
  The straightworard thing to do would be to either
  
  * resample inputs to a constant frequency
  * ignore input frequency and hope the RNN will figure it all out
  
  None sound very appealing. Is there a better way to handle variable input frequency in RNNs?
  
  **Tags:** recurrent-neural-networks
  > **Comments:**
  > **username_1**: Could you be more specific about the nature of the application, because I don't think there is a general answer here? For instance, You cannot realistically expect to perform certain tasks on video at 1fps regardless of the analysis technology - e.g. try to estimate speed of object moving through the view when only one frame captures that it was there in the first place.  
  > **username_0**: @NeilSlater For audio, it could be a speech-to-text task. For video, you could imagine to offer an action detection software solution that would not enforce an fps to the client. Obviously you would expect that a lower framerate degrades the precision of the predictions -- otherwise you might just as well resample all records to 1fps.  
  > **username_1**: There is "variation" and then there is "completely different problem".I cannot give a full answer here (not enough experience with RNNs), but would actually expect a difference of over an order of magnitude in sampling rates is going to have such a deleterious effect that you will want more than one solution, and not attempt to build a single application. Smaller changes, or dealing with an over-abundance of data (e.g. 120fps in 4K resolution from a broadcast-quality camera, where you would downsample just to reduce processing costs) are probably OK.  
  
  ---
  
  ## 0 answers
  
  