<p>I have a task which I would like to teach an AI to perform. The input to the task will a screenshot of the screen and the output at any given time step is one of the following actions:</p>
<pre><code>    DoNothing
    Click(x, y)
    Drag(from_x, from_y, to_x, to_y)
    KeyPress(Key)
    LongKeyPress(Key, duration)

where:
    Key ∈ { 1 , 2 , 3 , 4 , Space , Up , Down , Left , Right }
</code></pre>
<p>I think that deciding which action to take could be done with a softmax output layer, but I don't know how I would also encode the parameters of each action. For example, the <code>Click</code> action also 2 co-ordinate values but the <code>Drag</code> action needs 4 co-ordinates.</p>
<p>My best idea at the moment is that I could use one neural network to decide on which action type to perform and for each action, train another network to decide the parameters. So for example I have the main network which decides to take a <code>Click</code> action, and then a secondary network which will choose the <code>x</code> and <code>y</code> co-ordinates (given the same input).</p>
<p>The problem with this is that I would end up training 6 networks (decide the action, <code>Click</code> co-ordinates, <code>Drag</code> co-ordinates, <code>KeyPress</code> key, <code>LongKeyPress</code> key, <code>LongKeyPress</code> duration). The drawback for this is that I would need a very large dataset with a lot of samples for each type.</p>
<p>I considered how it might be possible to encode all the values into one output, but I wouldn't know how to deal with missing values (for samples where the user clicked, what value should i use for the <code>Key</code> output of <code>KeyPress</code>).</p>
<p>I have tried to find research papers on the topic, but I must be missing the right keywords because I can't find anything about strategies for this problem. Any ideas or pointers to relevant material would be great!</p>

CC BY-SA 4.0
2022-05-14T07:59:31.437Z
I think your problem would fit to reinforcement learning, but you also mentioned a dataset - so the main question is: Given a screenshot, is there one and only one correct action to perform? If yes, I think you are on the right track with classification, If No you should explore the realm of [reinforcement learning (RL)](https://en.wikipedia.org/wiki/Reinforcement_learning).
Also, if this is RL, the number of possible actions is enormous (especially the continuous actions click + drag & drop) which makes the task really hard and you might want to simplify the actions a bit.
Chillston
CC BY-SA 4.0
2022-05-14T12:33:01.177Z
Unfortunately, I cannot explore the possibility of RL because I do not have a test environment in which the agent can 'practice'. Besides this, I will mention that I am not interested in finding the *most efficient* solution, but instead the *most human-like* solution. That is why I would like the agent to learn from a dataset of actions made by a human. I know that I might also be able to use a GAN architecture for this but that is something I have no done before. Also I am aware that the action space is huge, but I cannot see a feasible way to reduce it without also reducing expressiveness.
turbo_sheep
<neural-networks><game-ai>
Encoding Actions with Parameters in Neural Network Output
turbo_sheep
# Encoding Actions with Parameters in Neural Network Output

(Asked by: username_0 on 2022-05-14)

I have a task which I would like to teach an AI to perform. The input to the task will a screenshot of the screen and the output at any given time step is one of the following actions:
    DoNothing
    Click(x, y)
    Drag(from_x, from_y, to_x, to_y)
    KeyPress(Key)
    LongKeyPress(Key, duration)

where:
    Key ∈ { 1 , 2 , 3 , 4 , Space , Up , Down , Left , Right }

I think that deciding which action to take could be done with a softmax output layer, but I don't know how I would also encode the parameters of each action. For example, the Click action also 2 co-ordinate values but the Drag action needs 4 co-ordinates.
My best idea at the moment is that I could use one neural network to decide on which action type to perform and for each action, train another network to decide the parameters. So for example I have the main network which decides to take a Click action, and then a secondary network which will choose the x and y co-ordinates (given the same input).
The problem with this is that I would end up training 6 networks (decide the action, Click co-ordinates, Drag co-ordinates, KeyPress key, LongKeyPress key, LongKeyPress duration). The drawback for this is that I would need a very large dataset with a lot of samples for each type.
I considered how it might be possible to encode all the values into one output, but I wouldn't know how to deal with missing values (for samples where the user clicked, what value should i use for the Key output of KeyPress).
I have tried to find research papers on the topic, but I must be missing the right keywords because I can't find anything about strategies for this problem. Any ideas or pointers to relevant material would be great!

**Tags:** neural-networks, game-ai

> **Comments:**
> **username_1**: I think your problem would fit to reinforcement learning, but you also mentioned a dataset - so the main question is: Given a screenshot, is there one and only one correct action to perform? If yes, I think you are on the right track with classification, If No you should explore the realm of [reinforcement learning (RL)](https://en.wikipedia.org/wiki/Reinforcement_learning). Also, if this is RL, the number of possible actions is enormous (especially the continuous actions click + drag & drop) which makes the task really hard and you might want to simplify the actions a bit.  
> **username_0**: Unfortunately, I cannot explore the possibility of RL because I do not have a test environment in which the agent can 'practice'. Besides this, I will mention that I am not interested in finding the *most efficient* solution, but instead the *most human-like* solution. That is why I would like the agent to learn from a dataset of actions made by a human. I know that I might also be able to use a GAN architecture for this but that is something I have no done before. Also I am aware that the action space is huge, but I cannot see a feasible way to reduce it without also reducing expressiveness.  

---

## 0 answers

