Answers.Body: <p>This is an old area of AI called "<a href="https://scholar.google.com/scholar?hl=en&amp;as_sdt=0%2C33&amp;authuser=1&amp;q=plan%20recognition&amp;btnG=&amp;oq=Plan%20Rec" rel="nofollow noreferrer">Plan Recognition</a>", which has about 3.5 million results in Google Scholar.</p>
  
  <p>A lot of the modern work is done with <a href="https://www.ijcai.org/proceedings/2018/0668.pdf" rel="nofollow noreferrer">classical search techniques</a> coupled with <a href="https://www.ijcai.org/proceedings/2018/0864.pdf" rel="nofollow noreferrer">expert domain knowledge</a>, or related reasoning concepts like <a href="http://www.planrec.org/PAIR/PAIR18/Papers/HollerPair18.pdf" rel="nofollow noreferrer">Hierarchical Task Networks</a>.</p>
  
  <p>I'm not aware of or able to find recent research using deep neural networks for this problem, but I think there are some data-drive approaches to the related work in video-game player modeling.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-23T21:56:01.937Z
Answers.LastActivityDate: 2018-07-23T21:56:01.937Z
Answers.UserDisplayName: John Doucette
Body: <p>The ability to recognize an object with particular identifying features from single or multiple camera shoots with the temporal dimension digitized as frames has been shown.  The proof is that the movie industry does face replacement to reduce liability costs for stars when stunts are needed.  It is now done in a substantial percentage of action movie releases.</p>
  
  <p>This brings up the question of how valuable recognizing a stop sign is compared to the value of recognizing an action.  For instance, in the world of autonomous vehicles, should there even be stop signs.  Stop signs are designed for lack of intelligence or lack of attention, which is why any police officer will tell you that almost no one comes to a full stop per law.  What human brains intuitively looks for is the potential of collision.</p>
  
  <p>Once what we linguistically perceive as verbs can be handled in deep learning scenarios as proficiently as nouns can be handled, the projection of risk becomes possible.</p>
  
  <p>This may be very much the philosophy behind the proprietary technology that allows directors to say, "Replace the stunt person's face with the movie's protagonist's face," and have a body of experts execute it using software tools and LINUX clusters.  The star's face is projected into the model of the action realized in the digital record of the stunt person.</p>
  
  <p>Projected action is exactly what our brain does when we avoid collisions, and not just with driving.  We do it socially, financially, when we design mechanical mechanisms, and in hundreds of other fields of human endeavor.</p>
  
  <p>If we consider the topology of GANs as a loop in balance, which is what it is, we can then see the similarity of GANs to the chemical equilibria between suspensions and solutions.  This gives us a hint into the type of topologies that can project action and therefore detect risk from audiovisual data streams.</p>
  
  <p>Once action recognition is mastered, it is a smaller step to use the trained model to project the next set of frames and then detect collision or other risks.  Such would most likely make possible a more reliable and safe automation of a number of AI products and services, breaking through a threshold in ML, and increased safety margins throughout the ever increasing world population density.</p>
  
  <p>... which brings us back to ...</p>
  
  <p><strong>What topologies support recognition of action sequences?</strong></p>
  
  <p>The topology may have convolution, perhaps in conjunction with RNN techniques, encoders, equilibria such as the generative and discriminative models in GANs, and other design elements and concepts.  Perhaps a new element type or concept will need to be invented.  Will we have to first recognize actions in a frame sequence and then project the consequences of various options in frames that are not yet shot?</p>
  
  <p>Where would the building blocks go and how would they be connected, initially dismissing concerns about computing power, network realization, and throughput for now?</p>
  
  <p>Work may have been done along this area and realized in software, but I have not seen that degree of maturity yet in the literature, so most of it, if there is any, must be proprietary at this time.  It is useful to open the question to the AI community and level the playing field.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-25T10:58:51.020Z
Comments.Text: I don't think it's clear what you're asking. Is it about classification (given footage of full sequence of actions, correctly recognize the full sequence), or prediction / modelling the world (given footage of part of the sequence, predict what the remainder of the sequence is)? The paragraphs referencing movies seem vague, and there's no evidence that stuff isn't simply done manual editing. I don't see how the paragraph about GANs / chemistry is relevant. Is the question about current state of the art, or more speculative in nature? It is not clear what is meant with "topologies".
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-25T11:37:03.820Z
Comments.Text: At the very least, we should know if you're asking about classification or prediction as mentioned in my previous comment, so that there is at least a well-defined question if we manage to dig through all the terminology. If you're worried about the answer becoming "obvious", [it is perfectly fine to directly answer your own question](https://stackoverflow.com/help/self-answer) (using the same account you used to post the question, not using a different account)
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-25T12:14:33.353Z
Comments.Text: @DennisSoemers, It is both explicit and clear to ask, "What topologies support recognition of action sequences?" and, "Where would the building blocks go and how would they be connected?" In aeronautics, fuel economy and maneuverability led to the definition of a key question: "What topologies provide high propulsive force to weight ratios? Where would the building blocks go and how would they be connected?" Pistons and valves gave way to compressors and turbines and jet engines were born. General action recognition is similarly important.
Comments.UserDisplayName: Douglas Daseeco
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-25T12:20:44.660Z
Comments.Text: @FauChristian Look back at my very first comment. I described there how "recognition of action sequences" can be interpreted in two very different ways. It is not clear from the question which interpretation (or maybe both) is intended, and that's crucial to know for a good answer. The word "topologies" is still not well-defined, so that also makes the question vague.
Comments.UserDisplayName: Dennis Soemers
Tags: <machine-learning><ai-safety><action-recognition><topology><autonomous-vehicles>
Title: What topologies support recognition of action sequences?
UserDisplayName: Douglas Daseeco
ThreadText: # What topologies support recognition of action sequences?
  
  (Asked by: username_0 on 2018-07-23)
  
  The ability to recognize an object with particular identifying features from single or multiple camera shoots with the temporal dimension digitized as frames has been shown. The proof is that the movie industry does face replacement to reduce liability costs for stars when stunts are needed. It is now done in a substantial percentage of action movie releases.
  
  This brings up the question of how valuable recognizing a stop sign is compared to the value of recognizing an action. For instance, in the world of autonomous vehicles, should there even be stop signs. Stop signs are designed for lack of intelligence or lack of attention, which is why any police officer will tell you that almost no one comes to a full stop per law. What human brains intuitively looks for is the potential of collision.
  
  Once what we linguistically perceive as verbs can be handled in deep learning scenarios as proficiently as nouns can be handled, the projection of risk becomes possible.
  
  This may be very much the philosophy behind the proprietary technology that allows directors to say, "Replace the stunt person's face with the movie's protagonist's face," and have a body of experts execute it using software tools and LINUX clusters. The star's face is projected into the model of the action realized in the digital record of the stunt person.
  
  Projected action is exactly what our brain does when we avoid collisions, and not just with driving. We do it socially, financially, when we design mechanical mechanisms, and in hundreds of other fields of human endeavor.
  
  If we consider the topology of GANs as a loop in balance, which is what it is, we can then see the similarity of GANs to the chemical equilibria between suspensions and solutions. This gives us a hint into the type of topologies that can project action and therefore detect risk from audiovisual data streams.
  
  Once action recognition is mastered, it is a smaller step to use the trained model to project the next set of frames and then detect collision or other risks. Such would most likely make possible a more reliable and safe automation of a number of AI products and services, breaking through a threshold in ML, and increased safety margins throughout the ever increasing world population density.
  
  ... which brings us back to ...
  
  **What topologies support recognition of action sequences?**
  
  The topology may have convolution, perhaps in conjunction with RNN techniques, encoders, equilibria such as the generative and discriminative models in GANs, and other design elements and concepts. Perhaps a new element type or concept will need to be invented. Will we have to first recognize actions in a frame sequence and then project the consequences of various options in frames that are not yet shot?
  
  Where would the building blocks go and how would they be connected, initially dismissing concerns about computing power, network realization, and throughput for now?
  
  Work may have been done along this area and realized in software, but I have not seen that degree of maturity yet in the literature, so most of it, if there is any, must be proprietary at this time. It is useful to open the question to the AI community and level the playing field.
  
  **Tags:** machine-learning, ai-safety, action-recognition, topology, autonomous-vehicles
  
  > **Comments:**
  > **username_1**: I don't think it's clear what you're asking. Is it about classification (given footage of full sequence of actions, correctly recognize the full sequence), or prediction / modelling the world (given footage of part of the sequence, predict what the remainder of the sequence is)? The paragraphs referencing movies seem vague, and there's no evidence that stuff isn't simply done manual editing. I don't see how the paragraph about GANs / chemistry is relevant. Is the question about current state of the art, or more speculative in nature? It is not clear what is meant with "topologies".  
  > **username_1**: At the very least, we should know if you're asking about classification or prediction as mentioned in my previous comment, so that there is at least a well-defined question if we manage to dig through all the terminology. If you're worried about the answer becoming "obvious", [it is perfectly fine to directly answer your own question](https://stackoverflow.com/help/self-answer) (using the same account you used to post the question, not using a different account)  
  > **username_0**: @username_1, It is both explicit and clear to ask, "What topologies support recognition of action sequences?" and, "Where would the building blocks go and how would they be connected?" In aeronautics, fuel economy and maneuverability led to the definition of a key question: "What topologies provide high propulsive force to weight ratios? Where would the building blocks go and how would they be connected?" Pistons and valves gave way to compressors and turbines and jet engines were born. General action recognition is similarly important.  
  > **username_1**: @FauChristian Look back at my very first comment. I described there how "recognition of action sequences" can be interpreted in two very different ways. It is not clear from the question which interpretation (or maybe both) is intended, and that's crucial to know for a good answer. The word "topologies" is still not well-defined, so that also makes the question vague.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-07-23. Score: 1
  
  This is an old area of AI called "[Plan Recognition](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C33&authuser=1&q=plan%20recognition&btnG=&oq=Plan%20Rec)", which has about 3.5 million results in Google Scholar.
  
  A lot of the modern work is done with [classical search techniques](https://www.ijcai.org/proceedings/2018/0668.pdf) coupled with [expert domain knowledge](https://www.ijcai.org/proceedings/2018/0864.pdf), or related reasoning concepts like [Hierarchical Task Networks](http://www.planrec.org/PAIR/PAIR18/Papers/HollerPair18.pdf).
  
  I'm not aware of or able to find recent research using deep neural networks for this problem, but I think there are some data-drive approaches to the related work in video-game player modeling.
  
  
  ---
  
  