Body: <p>I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image under construction to resemble the ideal image more. So basically, it translates image differences into motor commands to make them more similar?</p>
  
  <p>I'm dealing with a 3D virtual environment, so the images are snapshots of objects and motor commands are simple alterations to the 3D shape.</p>
  
  <p>Probably the network needs two pre-trained CNNs with the same weights that extract image features, then output and concatenate those into a dense layer (or two), which converges into action-space. Training should probably happen via reinforcement learning</p>
  
  <p>Additionally, in the end it needs recurrence, since there are multiple motor actions it needs to do in a row to get closer to the intended result.</p>
  
  <p>Would there be any serious difficulties with this? Or are there any approaches to achieve the intended result? or any similar examples?</p>
  
  <p>Thanks in advance</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T10:33:20.980Z
Comments.Text: Is there an obvious/detectable endpoint for a sequence of actions (e.g getting within some limit of ideal), or an enforced one (such as max number of steps) - or both, or is this issue open in your design?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T10:38:53.723Z
Comments.Text: This is still an open issue in my design. Probably starting more basic, with an enforced one of a few steps to see how it performs
Comments.UserDisplayName: SumakuTension
Tags: <neural-networks><reinforcement-learning><convolutional-neural-networks><computer-vision><recurrent-neural-networks>
Title: Ideas on a network that can translate image differences into motor commands?
UserDisplayName: SumakuTension
ThreadText: # Ideas on a network that can translate image differences into motor commands?
  
  (Asked by: username_0 on 2019-11-11)
  
  I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image under construction to resemble the ideal image more. So basically, it translates image differences into motor commands to make them more similar?
  
  I'm dealing with a 3D virtual environment, so the images are snapshots of objects and motor commands are simple alterations to the 3D shape.
  
  Probably the network needs two pre-trained CNNs with the same weights that extract image features, then output and concatenate those into a dense layer (or two), which converges into action-space. Training should probably happen via reinforcement learning
  
  Additionally, in the end it needs recurrence, since there are multiple motor actions it needs to do in a row to get closer to the intended result.
  
  Would there be any serious difficulties with this? Or are there any approaches to achieve the intended result? or any similar examples?
  
  Thanks in advance
  
  **Tags:** neural-networks, reinforcement-learning, convolutional-neural-networks, computer-vision, recurrent-neural-networks
  
  > **Comments:**
  > **username_1**: Is there an obvious/detectable endpoint for a sequence of actions (e.g getting within some limit of ideal), or an enforced one (such as max number of steps) - or both, or is this issue open in your design?  
  > **username_0**: This is still an open issue in my design. Probably starting more basic, with an enforced one of a few steps to see how it performs  
  
  ---
  
  ## 0 answers
  
  