Body: <p>I have a task which I would like to teach an AI to perform. The input to the task will a screenshot of the screen and the output at any given time step is one of the following actions:</p>
  <pre><code>    DoNothing
      Click(x, y)
      Drag(from_x, from_y, to_x, to_y)
      KeyPress(Key)
      LongKeyPress(Key, duration)
  
  where:
      Key ∈ { 1 , 2 , 3 , 4 , Space , Up , Down , Left , Right }
  </code></pre>
  <p>I think that deciding which action to take could be done with a softmax output layer, but I don't know how I would also encode the parameters of each action. For example, the <code>Click</code> action also 2 co-ordinate values but the <code>Drag</code> action needs 4 co-ordinates.</p>
  <p>My best idea at the moment is that I could use one neural network to decide on which action type to perform and for each action, train another network to decide the parameters. So for example I have the main network which decides to take a <code>Click</code> action, and then a secondary network which will choose the <code>x</code> and <code>y</code> co-ordinates (given the same input).</p>
  <p>The problem with this is that I would end up training 6 networks (decide the action, <code>Click</code> co-ordinates, <code>Drag</code> co-ordinates, <code>KeyPress</code> key, <code>LongKeyPress</code> key, <code>LongKeyPress</code> duration). The drawback for this is that I would need a very large dataset with a lot of samples for each type.</p>
  <p>I considered how it might be possible to encode all the values into one output, but I wouldn't know how to deal with missing values (for samples where the user clicked, what value should i use for the <code>Key</code> output of <code>KeyPress</code>).</p>
  <p>I have tried to find research papers on the topic, but I must be missing the right keywords because I can't find anything about strategies for this problem. Any ideas or pointers to relevant material would be great!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-14T07:59:31.437Z
Comments.Text: I think your problem would fit to reinforcement learning, but you also mentioned a dataset - so the main question is: Given a screenshot, is there one and only one correct action to perform? If yes, I think you are on the right track with classification, If No you should explore the realm of [reinforcement learning (RL)](https://en.wikipedia.org/wiki/Reinforcement_learning).
  Also, if this is RL, the number of possible actions is enormous (especially the continuous actions click + drag & drop) which makes the task really hard and you might want to simplify the actions a bit.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-14T12:33:01.177Z
Comments.Text: Unfortunately, I cannot explore the possibility of RL because I do not have a test environment in which the agent can 'practice'. Besides this, I will mention that I am not interested in finding the *most efficient* solution, but instead the *most human-like* solution. That is why I would like the agent to learn from a dataset of actions made by a human. I know that I might also be able to use a GAN architecture for this but that is something I have no done before. Also I am aware that the action space is huge, but I cannot see a feasible way to reduce it without also reducing expressiveness.
Comments.UserDisplayName: turbo_sheep
Tags: <neural-networks><game-ai>
Title: Encoding Actions with Parameters in Neural Network Output
UserDisplayName: turbo_sheep
ThreadText: # Encoding Actions with Parameters in Neural Network Output
  
  (Asked by: username_0 on 2022-05-14)
  
  I have a task which I would like to teach an AI to perform. The input to the task will a screenshot of the screen and the output at any given time step is one of the following actions:
      DoNothing
      Click(x, y)
      Drag(from_x, from_y, to_x, to_y)
      KeyPress(Key)
      LongKeyPress(Key, duration)
  
  where:
      Key ∈ { 1 , 2 , 3 , 4 , Space , Up , Down , Left , Right }
  
  I think that deciding which action to take could be done with a softmax output layer, but I don't know how I would also encode the parameters of each action. For example, the Click action also 2 co-ordinate values but the Drag action needs 4 co-ordinates.
  My best idea at the moment is that I could use one neural network to decide on which action type to perform and for each action, train another network to decide the parameters. So for example I have the main network which decides to take a Click action, and then a secondary network which will choose the x and y co-ordinates (given the same input).
  The problem with this is that I would end up training 6 networks (decide the action, Click co-ordinates, Drag co-ordinates, KeyPress key, LongKeyPress key, LongKeyPress duration). The drawback for this is that I would need a very large dataset with a lot of samples for each type.
  I considered how it might be possible to encode all the values into one output, but I wouldn't know how to deal with missing values (for samples where the user clicked, what value should i use for the Key output of KeyPress).
  I have tried to find research papers on the topic, but I must be missing the right keywords because I can't find anything about strategies for this problem. Any ideas or pointers to relevant material would be great!
  
  **Tags:** neural-networks, game-ai
  > **Comments:**
  > **username_1**: I think your problem would fit to reinforcement learning, but you also mentioned a dataset - so the main question is: Given a screenshot, is there one and only one correct action to perform? If yes, I think you are on the right track with classification, If No you should explore the realm of [reinforcement learning (RL)](https://en.wikipedia.org/wiki/Reinforcement_learning). Also, if this is RL, the number of possible actions is enormous (especially the continuous actions click + drag & drop) which makes the task really hard and you might want to simplify the actions a bit.  
  > **username_0**: Unfortunately, I cannot explore the possibility of RL because I do not have a test environment in which the agent can 'practice'. Besides this, I will mention that I am not interested in finding the *most efficient* solution, but instead the *most human-like* solution. That is why I would like the agent to learn from a dataset of actions made by a human. I know that I might also be able to use a GAN architecture for this but that is something I have no done before. Also I am aware that the action space is huge, but I cannot see a feasible way to reduce it without also reducing expressiveness.  
  
  ---
  
  ## 0 answers
  
  