Answers.Body: <p>It seems you need a spatio-temporal model to learn human-body detection and action.
  With regards to interesting papers on the subject I would recommand to look at <a href="http://vicky.kalogeiton.info/" rel="nofollow noreferrer">Vicky Kalogeiton web site</a>. </p>
  
  <p>Her PhD thesis 
  <a href="http://calvin.inf.ed.ac.uk/wp-content/uploads/Publications/theses/Kalogeiton2017.pdf" rel="nofollow noreferrer">2017, V. Kalogeiton, Localizing spatially and temporally objects and actions in videos, </a></p>
  
  <p>basically cover her 3 papers on the subject:</p>
  
  <ul>
  <li><a href="https://arxiv.org/pdf/1501.01186.pdf" rel="nofollow noreferrer">2016, Kalogeiton, V., Schmid, C., and Ferrari, V. Analysing domain shift factors between videos and images for object detection</a>,  </li>
  <li>[2017a, Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. Action tubelet detector for spatio-temporal action localization]<a href="https://arxiv.org/abs/1705.01861" rel="nofollow noreferrer">Kalogeiton 2017a</a>,</li>
  <li><a href="https://hal.inria.fr/hal-01575804/document" rel="nofollow noreferrer">2017b, Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. Joint learning of object and action detectors</a></li>
  </ul>
  
  <h1>summary of Kalogeiton's PhD introduction .</h1>
  
  <p>Introduce an end-to-end multitask objective that jointly learns object-action relationships. 
  The action-object detector leverages the temporal continuity of videos. </p>
  
  <p>Though <strong>intra class variations</strong> are key and appears as spatial location accuracy, appearance diversity, image quality, aspect distribution, and object size and camera framing.<br>
  Actions class refers to an atomic class such as jump, walk, run, climb, etc. </p>
  
  <p>The detector builds <code>anchor cuboids</code> named <strong>tubelets</strong> and defined as sequences of bounding boxes with associated scores. The action detection spans over a period of time (first and last video frame detected) and takes place at a specific location in each frame. Intra frame action detection can be ambigous. On the other way a sequence bears more information (<strong>across class similarities</strong>) than a single frame to infer action.</p>
  
  <p>Most previous work uses per-frame object detections, and then leverage the motion of objects to refine their spatial localization or improve their classification.</p>
  
  <p>Contributions</p>
  
  <ul>
  <li>differences between still and video frames for training and testing an object detector among which (see Chapter 3 for more details ):
  
  <ul>
  <li>spatial location accuracy, </li>
  <li>appearance diversity,</li>
  <li>image quality, </li>
  <li>aspect distribution, </li>
  <li>camera framing</li>
  </ul></li>
  <li>jointly detect object-action instances in uncontrolled videos using an end-to-end two stream network architecture (see chapter 4 for more details )</li>
  <li>propose the <strong>ACtion Tubelet detector (ACT-detector)</strong>, which takes as input a sequence of frames and outputs <strong>tubelets</strong>, i.e. sequences of bounding boxes with associated scores (see chapter 5 for more details).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T11:56:03.970Z
Answers.Text: this answer could complement mine https://ai.stackexchange.com/questions/1481/how-can-action-recognition-be-achieved?noredirect=1&lq=1
Answers.UserDisplayName: pascal sautot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T10:40:39.480Z
Answers.LastActivityDate: 2019-12-12T10:46:55.693Z
Answers.LastEditDate: 2019-12-12T10:46:55.693Z
Answers.UserDisplayName: pascal sautot
Body: <p>Consider a fixed camera that records a given area. Three things can happen in this area: </p>
  
  <ul>
  <li>No action</li>
  <li>People performing action A</li>
  <li>People performing action B</li>
  </ul>
  
  <p>I want to train a model to detect when action B happens. A human observer could typically recognize action B even with a single frame, but it would be much easier with a short video (a few seconds at low FPS).</p>
  
  <p>What are the most suitable models for this task? I read <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42455.pdf" rel="nofollow noreferrer">this paper</a> where different types of fusion are performed in order to feed different frames to a CNN. Are there better alternatives?</p>
  
Tags: <convolutional-neural-networks><video-classification>
Title: Most suitable model for video classification with a fixed camera
UserDisplayName: firion
ThreadText: # Most suitable model for video classification with a fixed camera
  
  (Asked by: username_0 on 2019-12-12)
  
  Consider a fixed camera that records a given area. Three things can happen in this area:
  
  * No action
  * People performing action A
  * People performing action B
  
  I want to train a model to detect when action B happens. A human observer could typically recognize action B even with a single frame, but it would be much easier with a short video (a few seconds at low FPS).
  
  What are the most suitable models for this task? I read [this paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42455.pdf) where different types of fusion are performed in order to feed different frames to a CNN. Are there better alternatives?
  
  **Tags:** convolutional-neural-networks, video-classification
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-12. Score: 3
  
  It seems you need a spatio-temporal model to learn human-body detection and action.
  With regards to interesting papers on the subject I would recommand to look at [Vicky Kalogeiton web site](http://vicky.kalogeiton.info/).
  
  Her PhD thesis
  [2017, V. Kalogeiton, Localizing spatially and temporally objects and actions in videos,](http://calvin.inf.ed.ac.uk/wp-content/uploads/Publications/theses/Kalogeiton2017.pdf)
  
  basically cover her 3 papers on the subject:
  
  * [2016, Kalogeiton, V., Schmid, C., and Ferrari, V. Analysing domain shift factors between videos and images for object detection](https://arxiv.org/pdf/1501.01186.pdf),
  * [2017a, Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. Action tubelet detector for spatio-temporal action localization][Kalogeiton 2017a](https://arxiv.org/abs/1705.01861),
  * [2017b, Kalogeiton, V., Weinzaepfel, P., Ferrari, V., and Schmid, C. Joint learning of object and action detectors](https://hal.inria.fr/hal-01575804/document)
  
  # summary of Kalogeiton's PhD introduction .
  
  Introduce an end-to-end multitask objective that jointly learns object-action relationships.
  The action-object detector leverages the temporal continuity of videos.
  
  Though **intra class variations** are key and appears as spatial location accuracy, appearance diversity, image quality, aspect distribution, and object size and camera framing.  
  Actions class refers to an atomic class such as jump, walk, run, climb, etc.
  
  The detector builds `anchor cuboids` named **tubelets** and defined as sequences of bounding boxes with associated scores. The action detection spans over a period of time (first and last video frame detected) and takes place at a specific location in each frame. Intra frame action detection can be ambigous. On the other way a sequence bears more information (**across class similarities**) than a single frame to infer action.
  
  Most previous work uses per-frame object detections, and then leverage the motion of objects to refine their spatial localization or improve their classification.
  
  Contributions
  
  * differences between still and video frames for training and testing an object detector among which (see Chapter 3 for more details ):
    + spatial location accuracy,
    + appearance diversity,
    + image quality,
    + aspect distribution,
    + camera framing
  * jointly detect object-action instances in uncontrolled videos using an end-to-end two stream network architecture (see chapter 4 for more details )
  * propose the **ACtion Tubelet detector (ACT-detector)**, which takes as input a sequence of frames and outputs **tubelets**, i.e. sequences of bounding boxes with associated scores (see chapter 5 for more details).
  
  > **Comments:**
  > **username_1**: this answer could complement mine https://ai.stackexchange.com/questions/1481/how-can-action-recognition-be-achieved?noredirect=1&lq=1  
  
  ---
  
  