Answers.Body: <p>This <a href="http://dx.doi.org/10.1109/TPAMI.2012.59" rel="nofollow noreferrer">study from 2012</a> uses 3D <a href="https://ai.stackexchange.com/questions/tagged/conv-neural-network">convolutional neural networks (CNN)</a> for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. A very similar deep learning approach based on 3D CNN is demonstrated in the <a href="http://liris.cnrs.fr/Documents/Liris-5228.pdf" rel="nofollow noreferrer">LIRIS and Orange Labs study from 2011</a>.</p>
  
  <hr>
  
  <p>This <a href="https://www.robots.ox.ac.uk/~vgg/publications/2014/Simonyan14b/simonyan14b.pdf.pdf" rel="nofollow noreferrer">Oxford study from 2014</a> also uses a similar approach, but with two-stream CNN which incorporates spatial and temporal networks which can achieve good performance despite having limited training data. It recognises action from motion in the form of dense optical flow. For example:</p>
  
  <p><a href="https://i.sstatic.net/RWcT3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RWcT3.png" alt="Optical flow using ConvNets"></a></p>
  
  <hr>
  
  <p><a href="http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4224216" rel="nofollow noreferrer">Another study from 2007</a> demonstrates a method by detecting human falls based on a combination of motion history and human shape variation by analysing the video frames. It uses Motion History Image (MHI) to quantify the motion of the person.</p>
  
  <p><a href="https://i.sstatic.net/Hzn4z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Hzn4z.png" alt="Motion history image (MHI)"></a></p>
  
  <p><sup>Source: <a href="https://github.com/harishrithish7/Fall-Detection" rel="nofollow noreferrer">harishrithish7/Fall-Detection</a> at GitHub</sup></p>
  
  <hr>
  
  <p>An alternative general approach could be action detection based on the posture using DNN. See: <a href="https://ai.stackexchange.com/q/1644/8">How to achieve recognition of postures and gestures?</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-29T13:58:42.653Z
Answers.Text: 3D CNN is a great solution to the problem. but it also has it flows, 3D CNN with stereoscopic cameras will do it perfect, only issue we will face is the z depth and focus of the camera, the camera will need to focus on object x which will be driven by the 3D CNN SC system, i've being using this for the past view months, I've now decided that the public can try my method.
Answers.UserDisplayName: X3R0
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-19T16:51:02.167Z
Answers.LastActivityDate: 2016-08-24T08:48:23.077Z
Answers.LastEditDate: 2017-04-13T12:53:10.013Z
Answers.UserDisplayName: kenorb
Answers.Body: <p>MIT have done research  and implemented an incomplete version of action video recognition.</p>
  
  <p>With the use of MATLAB, NNetworks and a large set of training videos.</p>
  
  <p>My suggested set of comments on my previous answer indicate the usage of a multi interconnected NNet, verus MIT's image based NNet.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-12T17:24:47.190Z
Answers.LastActivityDate: 2016-08-12T17:24:47.190Z
Answers.UserDisplayName: X3R0
Answers.Body: <p>A neural network can be used but must be trained to expect the information (pattern of data, pixels or groupings of loose range such as color, and location) at any given location in the network, first a vision system must but implemented. Then a facial recognition, multiple partial individual body fixing (finding body part and there partners to a person) then training on some states and you'll have it work. MIT have done research and have made a seemy accurate implementation. </p>
  
  <p>I'm an AI Researcher and Software Engineer for the past 7 years.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-12T17:10:36.330Z
Answers.Text: Seemy accurate meaning an incomplete but working system
Answers.UserDisplayName: X3R0
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-12T17:11:53.033Z
Answers.Text: Do a google search regarding this, or look on MITs website under thier video section. For more information.
Answers.UserDisplayName: X3R0
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-12T17:06:48.000Z
Answers.LastActivityDate: 2016-08-12T20:00:30.747Z
Answers.LastEditDate: 2016-08-12T20:00:30.747Z
Answers.UserDisplayName: X3R0
Answers.Body: <p><strong>No General Movie Search Yet</strong></p>
  <p>There have been successes in recognizing a very narrow sequence of a very narrow set of possible actions, but nothing like a general movie searching system that can return a set of matches with the start time, end time, and movie instance for each match to one of the search criteria listed in this question.</p>
  <ul>
  <li>Somebody was driving a car</li>
  <li>Kissing</li>
  <li>Eating</li>
  <li>Scared</li>
  <li>Talking over the phone</li>
  </ul>
  <p><strong>Normalizing the List</strong></p>
  <p>First of all, &quot;Was scared,&quot; is not the description of an action.  It should be, &quot;Becoming scared.&quot;  Secondly, &quot;Talking over the phone,&quot; is not a proper action description.  It should be a conjunctive action such as, &quot;Talking into a phone <em>AND</em> listening to the same phone.&quot;  To make the list homogenous in format, the first item should be &quot;Car driving,&quot; since the actor is human in every other case.</p>
  <ul>
  <li>Car driving</li>
  <li>Kissing</li>
  <li>Eating</li>
  <li>Becoming scared</li>
  <li>Talking into a phone and listening to the same phone.</li>
  </ul>
  <p><strong>Realistic System Design Expectations</strong></p>
  <p>It is unrealistic to think that an artificial neural net, by itself, can be trained to return as output the set of start and stop ranges and associated movie instances from a database of movies and one of the above list items as input.  This will require a complex system with many ANNs and other ML devices and may require other AI components that are not activation type networks at all.  Certainly convolution kernels and various types of encoders should be considered as key system components.</p>
  <p>You will need a large amount of training data to cover the above six cases (the last of the five items actually being two distinct actions that we normally associate and consider one).  If you want to detect more actions, you will need a large amount of training data for them too.</p>
  <p><strong>Verbs and Nouns</strong></p>
  <p>The reason this question is interesting to me is that recognizing ACTIONS are not the same as recognizing ITEMS.  All mammals learn ITEMS first and ACTIONS later.  Linguistically, nouns come before verbs in child language development.  That is because, just as detecting edges is preliminary to detecting shapes, which is preliminary to detecting objects, detecting motion is preliminary to detecting action.</p>
  <p>Verbs like, &quot;Eating,&quot; are an abstraction over the top of the motion, and, in the case of eating, the motion is complex.  Also, eating is not the same thing as gum chewing, so the sequence detected must be as follows:</p>
  <ol>
  <li>Insertion of food into the face through the mouth</li>
  <li>Chewing</li>
  <li>Swallowing</li>
  </ol>
  <p>The probability of a sequence is the product of the probability of its parts so that math is simple and easy to implement.  Concurrency, as in the case of conjunctive actions like talking into and listening to the same phone, is also relatively easy to handle in general.</p>
  <p><strong>A Realistic Approach</strong></p>
  <p>Certainly, generalization (and more specifically feature extraction) will need to occur in object recognition, collision detection, motion detection, facial recognition, and other planes simultaneously.  A complex topology, perhaps employing equilibria as in GAN design, will most likely be necessary to assemble elements of criteria associated with the movie query string and to run windows over the frames of each movie.</p>
  <p>To provide a service that returns results within a few days or weeks will probably require a cluster and DSP hardware (perhaps leveraging GPUs).</p>
  <p><strong>Special Cases that Human Brains Handle</strong></p>
  <p>Determining how long one of the two elements of concurrency can be undetected before it invalidates the conjunction can be tricky.  (How long can one not speak into the phone before it appears that it is no longer considered phone conversation?)</p>
  <p>If in the movie, only the swallowing is shown, a human can infer eating.  That kind of conclusion reliability from sparse data is a huge AI challenge discussed in various contexts throughout the literature.</p>
  <p><strong>The Emergence of Associated Technology — A Projection</strong></p>
  <p>I suspect that the system topography comprised of ANNs, encoders, convolution kernels, and other components to perform the search for any of a select set of actions will emerge within the next ten years.  Work seems to be tracking in that direction in the literature.</p>
  <p>A system that will acquire its own training information, sustainably grow in knowledge and perform general searches if increasing breadth and complexity may be anywhere from forty to two hundred years out.  It is difficult to predict.</p>
  <p><strong>Gross Overoptimistic Predictions</strong></p>
  <p>Every generation seems to view knowledge growth as an exponential function and tends to make unrealistic predictions about the advent of certain coveted technology capabilities.  Most of the predictions fail dramatically.  I have come to believe that exponential growth is an illusion created by the inverse exponential decay of interest in the past with respect to time.</p>
  <p>We lose track of the energy and rate of growth in eras before us because they become socially irrelevant.  People into scientific history, like Whitehead, Kuhn, and Ellul know that technology has moved forward quickly for at least a few hundred years.  Vernadsky inferred in his <em>The Biosphere</em> that life may not have arisen, that like matter and energy, it may always have existed.  I wonder if technology has been moving at an essentially constant rate for the last 50,000 years.</p>
  <p>Germany decided to double its solar panel energy output every year and published its exponential success, until a few years ago when doubling it again would cost a hundred billion dollars more than what they had to spend.  They stopped publishing the exponential growth graphs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-24T13:12:28.507Z
Answers.LastActivityDate: 2021-03-31T22:21:17.730Z
Answers.LastEditDate: 2021-03-31T22:21:17.730Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>There are several approaches as to how this can be achieved.</p>
  
  <p>One recent study from 2015 about <a href="http://link.springer.com/chapter/10.1007%2F978-3-319-09396-3_9" rel="nofollow noreferrer">Action Recognition in Realistic Sports Videos</a><sup><a href="http://cs.stanford.edu/~amirz/index_files/Springer2015_action_chapter.pdf" rel="nofollow noreferrer">PDF</a></sup> uses the action recognition framework based on the three main steps of feature extraction (shape, post or contextual information), dictionary learning to represent a video, and classification (<a href="https://en.wikipedia.org/wiki/Bag-of-words_model" rel="nofollow noreferrer">BoW framework</a>). </p>
  
  <p>A few examples of methods:</p>
  
  <ul>
  <li><p>Spatio-Temporal Structures of Human Poses</p>
  
  <p><a href="https://i.sstatic.net/AgfDk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AgfDk.png" alt="K. Soomro and A.R. Zamir - action recognition - figure"></a></p></li>
  <li><p>a joint shape-motion</p>
  
  <p><a href="https://i.sstatic.net/1qR9x.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1qR9x.png" alt="K. Soomro and A.R. Zamir - action recognition - figure"></a></p></li>
  <li><p>Multi-Task Sparse Learning (MTSL)</p></li>
  <li><p>Hierarchical Space-Time Segments</p>
  
  <p><a href="https://i.sstatic.net/jBMuj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jBMuj.png" alt="K. Soomro and A.R. Zamir - Extracted segments from video frames"></a></p></li>
  <li><p>Spatio-Temporal Deformable Part Models (SDPM)</p>
  
  <p><a href="https://i.sstatic.net/5Zehk.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5Zehk.jpg" alt="K. Soomro and A.R. Zamir - Action localization results"></a></p></li>
  </ul>
  
  <p>Here are the results based on training of 10 action classes based on the UCF sports dataset:</p>
  
  <p><a href="https://i.sstatic.net/v77Pv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v77Pv.png" alt="UCF Sports Dataset: sample frames of 10 action classes along with their bounding box annotations of the humans shown in yellow"></a></p>
  
  <p><sup>Source: <a href="http://link.springer.com/chapter/10.1007%2F978-3-319-09396-3_9" rel="nofollow noreferrer">Action Recognition in Realistic Sports Videos</a>.</sup></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-19T16:52:09.607Z
Answers.LastActivityDate: 2016-08-24T08:26:21.633Z
Answers.LastEditDate: 2016-08-24T08:26:21.633Z
Answers.UserDisplayName: kenorb
Body: <p>For example, I would like to train my neural network to recognize the type of actions (e.g. in commercial movies or some real-life videos), so I can "ask" my network in which video or movie (and at what frames) somebody was driving a car, kissing, eating, was scared or was talking over the phone.</p>
  
  <p>What are the current successful approaches to that type of problem?</p>
  
Tags: <machine-learning><image-recognition><training><action-recognition>
Title: How can action recognition be achieved?
UserDisplayName: kenorb
ThreadText: # How can action recognition be achieved?
  
  (Asked by: username_0 on 2016-08-09)
  
  For example, I would like to train my neural network to recognize the type of actions (e.g. in commercial movies or some real-life videos), so I can "ask" my network in which video or movie (and at what frames) somebody was driving a car, kissing, eating, was scared or was talking over the phone.
  
  What are the current successful approaches to that type of problem?
  
  **Tags:** machine-learning, image-recognition, training, action-recognition
  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_0 on 2016-08-19. Score: 5 | Accepted answer
  
  There are several approaches as to how this can be achieved.
  
  One recent study from 2015 about [Action Recognition in Realistic Sports Videos](http://link.springer.com/chapter/10.1007%2F978-3-319-09396-3_9)[PDF](http://cs.stanford.edu/~amirz/index_files/Springer2015_action_chapter.pdf) uses the action recognition framework based on the three main steps of feature extraction (shape, post or contextual information), dictionary learning to represent a video, and classification ([BoW framework](https://en.wikipedia.org/wiki/Bag-of-words_model)).
  
  A few examples of methods:
  
  * Spatio-Temporal Structures of Human Poses
  
    [![K. Soomro and A.R. Zamir - action recognition - figure](https://i.sstatic.net/AgfDk.png)](https://i.sstatic.net/AgfDk.png)
  * a joint shape-motion
  
    [![K. Soomro and A.R. Zamir - action recognition - figure](https://i.sstatic.net/1qR9x.png)](https://i.sstatic.net/1qR9x.png)
  * Multi-Task Sparse Learning (MTSL)
  * Hierarchical Space-Time Segments
  
    [![K. Soomro and A.R. Zamir - Extracted segments from video frames](https://i.sstatic.net/jBMuj.png)](https://i.sstatic.net/jBMuj.png)
  * Spatio-Temporal Deformable Part Models (SDPM)
  
    [![K. Soomro and A.R. Zamir - Action localization results](https://i.sstatic.net/5Zehk.jpg)](https://i.sstatic.net/5Zehk.jpg)
  
  Here are the results based on training of 10 action classes based on the UCF sports dataset:
  
  [![UCF Sports Dataset: sample frames of 10 action classes along with their bounding box annotations of the humans shown in yellow](https://i.sstatic.net/v77Pv.png)](https://i.sstatic.net/v77Pv.png)
  
  Source: [Action Recognition in Realistic Sports Videos](http://link.springer.com/chapter/10.1007%2F978-3-319-09396-3_9).
  
  
  ---
  
  ### Answer by username_0 on 2016-08-19. Score: 5
  
  This [study from 2012](http://dx.doi.org/10.1109/TPAMI.2012.59) uses 3D [convolutional neural networks (CNN)](https://ai.stackexchange.com/questions/tagged/conv-neural-network) for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. A very similar deep learning approach based on 3D CNN is demonstrated in the [LIRIS and Orange Labs study from 2011](http://liris.cnrs.fr/Documents/Liris-5228.pdf).
  
  ---
  
  This [Oxford study from 2014](https://www.robots.ox.ac.uk/~vgg/publications/2014/Simonyan14b/simonyan14b.pdf.pdf) also uses a similar approach, but with two-stream CNN which incorporates spatial and temporal networks which can achieve good performance despite having limited training data. It recognises action from motion in the form of dense optical flow. For example:
  
  [![Optical flow using ConvNets](https://i.sstatic.net/RWcT3.png)](https://i.sstatic.net/RWcT3.png)
  
  ---
  
  [Another study from 2007](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4224216) demonstrates a method by detecting human falls based on a combination of motion history and human shape variation by analysing the video frames. It uses Motion History Image (MHI) to quantify the motion of the person.
  
  [![Motion history image (MHI)](https://i.sstatic.net/Hzn4z.png)](https://i.sstatic.net/Hzn4z.png)
  
  Source: [harishrithish7/Fall-Detection](https://github.com/harishrithish7/Fall-Detection) at GitHub
  
  ---
  
  An alternative general approach could be action detection based on the posture using DNN. See: [How to achieve recognition of postures and gestures?](https://ai.stackexchange.com/q/1644/8)
  
  > **Comments:**
  > **username_1**: 3D CNN is a great solution to the problem. but it also has it flows, 3D CNN with stereoscopic cameras will do it perfect, only issue we will face is the z depth and focus of the camera, the camera will need to focus on object x which will be driven by the 3D CNN SC system, i've being using this for the past view months, I've now decided that the public can try my method.  
  
  ---
  
  ### Answer by username_1 on 2016-08-12. Score: 3
  
  MIT have done research and implemented an incomplete version of action video recognition.
  
  With the use of MATLAB, NNetworks and a large set of training videos.
  
  My suggested set of comments on my previous answer indicate the usage of a multi interconnected NNet, verus MIT's image based NNet.
  
  
  ---
  
  ### Answer by username_1 on 2016-08-12. Score: 2
  
  A neural network can be used but must be trained to expect the information (pattern of data, pixels or groupings of loose range such as color, and location) at any given location in the network, first a vision system must but implemented. Then a facial recognition, multiple partial individual body fixing (finding body part and there partners to a person) then training on some states and you'll have it work. MIT have done research and have made a seemy accurate implementation.
  
  I'm an AI Researcher and Software Engineer for the past 7 years.
  
  > **Comments:**
  > **username_1**: Seemy accurate meaning an incomplete but working system  
  > **username_1**: Do a google search regarding this, or look on MITs website under thier video section. For more information.  
  
  ---
  
  ### Answer by username_2 on 2018-07-24. Score: 0
  
  **No General Movie Search Yet**
  
  There have been successes in recognizing a very narrow sequence of a very narrow set of possible actions, but nothing like a general movie searching system that can return a set of matches with the start time, end time, and movie instance for each match to one of the search criteria listed in this question.
  
  * Somebody was driving a car
  * Kissing
  * Eating
  * Scared
  * Talking over the phone
  
  **Normalizing the List**
  
  First of all, "Was scared," is not the description of an action. It should be, "Becoming scared." Secondly, "Talking over the phone," is not a proper action description. It should be a conjunctive action such as, "Talking into a phone *AND* listening to the same phone." To make the list homogenous in format, the first item should be "Car driving," since the actor is human in every other case.
  
  * Car driving
  * Kissing
  * Eating
  * Becoming scared
  * Talking into a phone and listening to the same phone.
  
  **Realistic System Design Expectations**
  
  It is unrealistic to think that an artificial neural net, by itself, can be trained to return as output the set of start and stop ranges and associated movie instances from a database of movies and one of the above list items as input. This will require a complex system with many ANNs and other ML devices and may require other AI components that are not activation type networks at all. Certainly convolution kernels and various types of encoders should be considered as key system components.
  
  You will need a large amount of training data to cover the above six cases (the last of the five items actually being two distinct actions that we normally associate and consider one). If you want to detect more actions, you will need a large amount of training data for them too.
  
  **Verbs and Nouns**
  
  The reason this question is interesting to me is that recognizing ACTIONS are not the same as recognizing ITEMS. All mammals learn ITEMS first and ACTIONS later. Linguistically, nouns come before verbs in child language development. That is because, just as detecting edges is preliminary to detecting shapes, which is preliminary to detecting objects, detecting motion is preliminary to detecting action.
  
  Verbs like, "Eating," are an abstraction over the top of the motion, and, in the case of eating, the motion is complex. Also, eating is not the same thing as gum chewing, so the sequence detected must be as follows:
  
  1. Insertion of food into the face through the mouth
  2. Chewing
  3. Swallowing
  
  The probability of a sequence is the product of the probability of its parts so that math is simple and easy to implement. Concurrency, as in the case of conjunctive actions like talking into and listening to the same phone, is also relatively easy to handle in general.
  
  **A Realistic Approach**
  
  Certainly, generalization (and more specifically feature extraction) will need to occur in object recognition, collision detection, motion detection, facial recognition, and other planes simultaneously. A complex topology, perhaps employing equilibria as in GAN design, will most likely be necessary to assemble elements of criteria associated with the movie query string and to run windows over the frames of each movie.
  
  To provide a service that returns results within a few days or weeks will probably require a cluster and DSP hardware (perhaps leveraging GPUs).
  
  **Special Cases that Human Brains Handle**
  
  Determining how long one of the two elements of concurrency can be undetected before it invalidates the conjunction can be tricky. (How long can one not speak into the phone before it appears that it is no longer considered phone conversation?)
  
  If in the movie, only the swallowing is shown, a human can infer eating. That kind of conclusion reliability from sparse data is a huge AI challenge discussed in various contexts throughout the literature.
  
  **The Emergence of Associated Technology — A Projection**
  
  I suspect that the system topography comprised of ANNs, encoders, convolution kernels, and other components to perform the search for any of a select set of actions will emerge within the next ten years. Work seems to be tracking in that direction in the literature.
  
  A system that will acquire its own training information, sustainably grow in knowledge and perform general searches if increasing breadth and complexity may be anywhere from forty to two hundred years out. It is difficult to predict.
  
  **Gross Overoptimistic Predictions**
  
  Every generation seems to view knowledge growth as an exponential function and tends to make unrealistic predictions about the advent of certain coveted technology capabilities. Most of the predictions fail dramatically. I have come to believe that exponential growth is an illusion created by the inverse exponential decay of interest in the past with respect to time.
  
  We lose track of the energy and rate of growth in eras before us because they become socially irrelevant. People into scientific history, like Whitehead, Kuhn, and Ellul know that technology has moved forward quickly for at least a few hundred years. Vernadsky inferred in his *The Biosphere* that life may not have arisen, that like matter and energy, it may always have existed. I wonder if technology has been moving at an essentially constant rate for the last 50,000 years.
  
  Germany decided to double its solar panel energy output every year and published its exponential success, until a few years ago when doubling it again would cost a hundred billion dollars more than what they had to spend. They stopped publishing the exponential growth graphs.
  
  
  ---
  
  