Answers.Body: <p>In the scenario as you present it, reinforcement learning (RL) should work, but you may gain very little by applying it, over simpler search algorithms.</p>
  <p>As you explain in comments, each application is separately trained, the task in each case is arbitrary, and there is no way for the agent to apply contextual knowledge to identify likely actions. The only feedback will be task completion. There are no heurtistics available, simply a list of available actions on each step.</p>
  <p>In terms of RL problem definition, this is very similar to solving a maze in least number of steps. The problem is episodic (it terminates), so setting a cost per action (negative reward e.g. <span class="math-container">$r = -1$</span> on each step) is the simplest way for the agent to get feedback on how it is doing. The highest total reward will represent the shortest path.</p>
  <p>Simple maze solvers are often used as toy problems in RL tutorials. You could use pretty much any example of a gridworld maze from such a tutorial as a starting point to make an agent for your problem, and replace the up/right/left/down actions with your more abstract list of text links or menu options. The state representation will depend on application behaviour, but could be as simple as the current list of available actions.</p>
  <p>For this final part, I am assuming the following things, implied by the question and comments:</p>
  <ul>
  <li>The application behaves completely deterministically.</li>
  <li>The entry point to the application is always the same.</li>
  <li>The output goal is a single, shortest-possible list of simple actions (labels to select from) that reach the goal from the start.</li>
  </ul>
  <p>If all the above are true, then RL is effectively a random search for the goal state followed by further random iterations to remove non-necessary steps and find the shortest path. In which case, other tree-searching algorithms may be simpler to implment and have better performance than RL. You could try depth-first search (DFS) and/or breadth-first search (BFS), perhaps also requiring loop detection/avoidance. These would avoid the randomness inherent in an RL-based search, and likely be far more efficient due to that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-09T08:13:45.047Z
Answers.Text: Thanks for the answer! Since I may gain very little by applying it, do you think in this scenario would be preferable a deep learning algorithm (maybe DQNs)?
  Or should I just stick with the OCR and computer vision approach?
Answers.UserDisplayName: zaxunobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-09T09:22:20.410Z
Answers.Text: @zaxunobi Deep learning will only help in similar scenarios to RL, when your automation scenarios have something in common and when something is known about the goal task when starting each one. The combination with OCR/vision is a different part of the puzzle. Potentially there is something worth doing there with ML. In the question you presented this as an existing component, but learning to click on meaningful components based on a screenshot might be learnable, separate to finding a correct path to end point
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-11T17:24:31.757Z
Answers.Text: Since the actions are pretty much clicking on buttons, what contextual knowledge could an agent apply to determine meaningful components (the action is always a click, what changes are the coordinates of the click)? 
  About the deep learning scenario, do you have in mind how it could learn to to click on meaningful components based on screenshots?
Answers.UserDisplayName: zaxunobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-11T18:23:58.263Z
Answers.Text: @zaxunobi I don't know enough about your goals and constraints to answer that, and definitely not in the comments. I find it odd that you are writing an automation tool without information on automation goals at the start, but only as a discoverable reward. If you want to explore options for deep RL, I suggest asking a new question giving more details of the project than you have so far
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-12T06:18:37.373Z
Answers.Text: The main goal is reaching a certain state of the program, or the "end" if you like.
  The reason why I am going with "discoverable reward" it's because the only information that I could provide it's basically (which I am doing with OCR already) is that if a certain text is detected then I am in a specific STATE and therefore will be performed a specific ACTION. But doing this would defeat the whole point of using RL, don't you think?
Answers.UserDisplayName: zaxunobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-20T07:35:15.053Z
Answers.Text: I've opened a new question: https://ai.stackexchange.com/questions/46033/is-deep-learning-suitable-preferable-for-string-similarity-detection-and-applica
Answers.UserDisplayName: zaxunobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-05T11:16:41.800Z
Answers.LastActivityDate: 2024-06-05T11:16:41.800Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have basically automatised the use of an app through the use of OCR and computer vision.</p>
  <p>So basically when a word or an image is detected it will perform a certain action.</p>
  <p>When that action is successfully completed it will go to the next state.</p>
  <p>Now I want to try basically with a more &quot;heuristic&quot; approach and I thought about reinforcement learning.
  Why? Because I am aiming to build a tool that basically understand automatically what actions to perform in a certain state.</p>
  <p>But I have a doubt.
  Even though I don't need to declare an association like this (it would beat the purpose of deep reinforcement learning or deep learning in general):</p>
  <pre><code>  if(state.MENU_VIEW)
      clickManager.clickOnFolder();
      ...
  </code></pre>
  <p>I still need to define the states, the actions and the reward.</p>
  <p>Meaning I would need to instruct my app that when the OCR result is &quot;Open Folder&quot; it means the state I am in is MENU_VIEW. I simply wouldn't tell my app what action to perform in a that state. Am I correct?</p>
  <p>What I am trying to say is: how exactly could I make it so that the states (and maybe also the actions?) are generated automatically?</p>
  <p>The reward in this case scenario would be basically the folder being opened successfully.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-31T07:29:17.043Z
Comments.Text: Welcome to AI Stack Exchange. It's better if the title is more self-contained (e.g. don't refer to "this scenario" and hope people will click through to read the whole body). I will edit it. If you don't like the edit, feel free to adjust further
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-31T07:32:25.430Z
Comments.Text: Could you clarify what is setting the goals of the automation? I.e. how are the multi-step goals decided, and communicated to the agent?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-31T12:56:31.683Z
Comments.Text: @NeilSlater thank your for the welcome and thank you for adjusting the title, as a matter of fact it was redundant.
Comments.UserDisplayName: zaxunobi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-31T13:10:26.570Z
Comments.Text: @NeilSlater In the example provided I could basically pass an int from the environment to the agent: -1 if the click is far from the folder icon, 0 is if near and 1 if it's exactly on top of the folder.
Comments.UserDisplayName: zaxunobi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-31T19:02:15.263Z
Comments.Text: That would seem like a reward signal - something you would send after the agent has taken an action. What I am asking is how does the agent know what its goal is? Is the goal _always_ to open a folder page, or is it more long term than that, and how flexible is the long term goal? I.e. are you trying to train an agent to achieve one specific task where the UI may vary (so it needs to learn where to look in different systems, but always to do the same thing), or are you trying to train the agent to undertake multiple similar tasks, and it needs to somehow infer which one?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T14:52:56.327Z
Comments.Text: @NeilSlater no, the folder page was just an example. Basically the goal is to reach the end of the application. 
  Let's say you have a series of dialogs or pages, the long-term goal is basically clicking on all the "right" buttons until you reach the last page which will display "Operation completed". A sort of visiting problem so to speak. For now I simplified the problem since there are instances in which a simple click might not be enough and are required multiple clicks to go to next state.
Comments.UserDisplayName: zaxunobi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T16:27:57.180Z
Comments.Text: Is the agent aware of the nature of the required operation when it starts, or is it trying all menu options that might lead to "the end" until it is successful? Does it get *any* clue on what it is attempting to do other than make the end message appear *somehow*?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-04T13:53:46.687Z
Comments.Text: Well, that is exactly what I am not so sure. 
  In my program, as I said, I would basically use OCR to detect the "right path".
  I've seen a reinforcement learning project of the game snake and the required operation, at first, was indeed random between RIGHT, LEFT, UP and DOWN.
  So, in my case I thought about clicking randomly on the menu options that might lead to the end.
  No, it doesn't get any more clues other than end message. At least so far I haven't thought about any more clues. Maybe I am still having a thought pattern that revolves around OCR.
Comments.UserDisplayName: zaxunobi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-04T15:52:09.930Z
Comments.Text: More questions, sorry. But this is relevant: Will the trained agent be used on exactly the same or sites it was trained on, or is it attempting to learn a strategy that could work on multiple sites (despite not knowing what its goal is)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-05T08:29:13.210Z
Comments.Text: @NeilSlater you're absolutely right, I should have mentioned it. The trained agent will be used exactly on the same site it was trained on.
Comments.UserDisplayName: zaxunobi
Tags: <reinforcement-learning><deep-learning><deep-rl>
Title: Is reinforcement learning suitable for application automation?
UserDisplayName: zaxunobi
ThreadText: # Is reinforcement learning suitable for application automation?
  
  (Asked by: username_0 on 2024-05-30)
  
  I have basically automatised the use of an app through the use of OCR and computer vision.
  So basically when a word or an image is detected it will perform a certain action.
  When that action is successfully completed it will go to the next state.
  Now I want to try basically with a more "heuristic" approach and I thought about reinforcement learning.
  Why? Because I am aiming to build a tool that basically understand automatically what actions to perform in a certain state.
  But I have a doubt.
  Even though I don't need to declare an association like this (it would beat the purpose of deep reinforcement learning or deep learning in general):
    if(state.MENU_VIEW)
      clickManager.clickOnFolder();
      ...
  
  I still need to define the states, the actions and the reward.
  Meaning I would need to instruct my app that when the OCR result is "Open Folder" it means the state I am in is MENU_VIEW. I simply wouldn't tell my app what action to perform in a that state. Am I correct?
  What I am trying to say is: how exactly could I make it so that the states (and maybe also the actions?) are generated automatically?
  The reward in this case scenario would be basically the folder being opened successfully.
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl
  > **Comments:**
  > **username_1**: Welcome to AI Stack Exchange. It's better if the title is more self-contained (e.g. don't refer to "this scenario" and hope people will click through to read the whole body). I will edit it. If you don't like the edit, feel free to adjust further  
  > **username_1**: Could you clarify what is setting the goals of the automation? I.e. how are the multi-step goals decided, and communicated to the agent?  
  > **username_0**: @NeilSlater thank your for the welcome and thank you for adjusting the title, as a matter of fact it was redundant.  
  > **username_0**: @NeilSlater In the example provided I could basically pass an int from the environment to the agent: -1 if the click is far from the folder icon, 0 is if near and 1 if it's exactly on top of the folder.  
  > **username_1**: That would seem like a reward signal - something you would send after the agent has taken an action. What I am asking is how does the agent know what its goal is? Is the goal _always_ to open a folder page, or is it more long term than that, and how flexible is the long term goal? I.e. are you trying to train an agent to achieve one specific task where the UI may vary (so it needs to learn where to look in different systems, but always to do the same thing), or are you trying to train the agent to undertake multiple similar tasks, and it needs to somehow infer which one?  
  > **username_0**: @NeilSlater no, the folder page was just an example. Basically the goal is to reach the end of the application.  Let's say you have a series of dialogs or pages, the long-term goal is basically clicking on all the "right" buttons until you reach the last page which will display "Operation completed". A sort of visiting problem so to speak. For now I simplified the problem since there are instances in which a simple click might not be enough and are required multiple clicks to go to next state.  
  > **username_1**: Is the agent aware of the nature of the required operation when it starts, or is it trying all menu options that might lead to "the end" until it is successful? Does it get *any* clue on what it is attempting to do other than make the end message appear *somehow*?  
  > **username_0**: Well, that is exactly what I am not so sure.  In my program, as I said, I would basically use OCR to detect the "right path". I've seen a reinforcement learning project of the game snake and the required operation, at first, was indeed random between RIGHT, LEFT, UP and DOWN. So, in my case I thought about clicking randomly on the menu options that might lead to the end. No, it doesn't get any more clues other than end message. At least so far I haven't thought about any more clues. Maybe I am still having a thought pattern that revolves around OCR.  
  > **username_1**: More questions, sorry. But this is relevant: Will the trained agent be used on exactly the same or sites it was trained on, or is it attempting to learn a strategy that could work on multiple sites (despite not knowing what its goal is)?  
  > **username_0**: @NeilSlater you're absolutely right, I should have mentioned it. The trained agent will be used exactly on the same site it was trained on.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-05. Score: 0
  
  In the scenario as you present it, reinforcement learning (RL) should work, but you may gain very little by applying it, over simpler search algorithms.
  
  As you explain in comments, each application is separately trained, the task in each case is arbitrary, and there is no way for the agent to apply contextual knowledge to identify likely actions. The only feedback will be task completion. There are no heurtistics available, simply a list of available actions on each step.
  
  In terms of RL problem definition, this is very similar to solving a maze in least number of steps. The problem is episodic (it terminates), so setting a cost per action (negative reward e.g. $r = -1$ on each step) is the simplest way for the agent to get feedback on how it is doing. The highest total reward will represent the shortest path.
  
  Simple maze solvers are often used as toy problems in RL tutorials. You could use pretty much any example of a gridworld maze from such a tutorial as a starting point to make an agent for your problem, and replace the up/right/left/down actions with your more abstract list of text links or menu options. The state representation will depend on application behaviour, but could be as simple as the current list of available actions.
  
  For this final part, I am assuming the following things, implied by the question and comments:
  
  * The application behaves completely deterministically.
  * The entry point to the application is always the same.
  * The output goal is a single, shortest-possible list of simple actions (labels to select from) that reach the goal from the start.
  
  If all the above are true, then RL is effectively a random search for the goal state followed by further random iterations to remove non-necessary steps and find the shortest path. In which case, other tree-searching algorithms may be simpler to implment and have better performance than RL. You could try depth-first search (DFS) and/or breadth-first search (BFS), perhaps also requiring loop detection/avoidance. These would avoid the randomness inherent in an RL-based search, and likely be far more efficient due to that.
  
  > **Comments:**
  > **username_0**: Thanks for the answer! Since I may gain very little by applying it, do you think in this scenario would be preferable a deep learning algorithm (maybe DQNs)? Or should I just stick with the OCR and computer vision approach?  
  > **username_1**: @zaxunobi Deep learning will only help in similar scenarios to RL, when your automation scenarios have something in common and when something is known about the goal task when starting each one. The combination with OCR/vision is a different part of the puzzle. Potentially there is something worth doing there with ML. In the question you presented this as an existing component, but learning to click on meaningful components based on a screenshot might be learnable, separate to finding a correct path to end point  
  > **username_0**: Since the actions are pretty much clicking on buttons, what contextual knowledge could an agent apply to determine meaningful components (the action is always a click, what changes are the coordinates of the click)?  About the deep learning scenario, do you have in mind how it could learn to to click on meaningful components based on screenshots?  
  > **username_1**: @zaxunobi I don't know enough about your goals and constraints to answer that, and definitely not in the comments. I find it odd that you are writing an automation tool without information on automation goals at the start, but only as a discoverable reward. If you want to explore options for deep RL, I suggest asking a new question giving more details of the project than you have so far  
  > **username_0**: The main goal is reaching a certain state of the program, or the "end" if you like. The reason why I am going with "discoverable reward" it's because the only information that I could provide it's basically (which I am doing with OCR already) is that if a certain text is detected then I am in a specific STATE and therefore will be performed a specific ACTION. But doing this would defeat the whole point of using RL, don't you think?  
  > **username_0**: I've opened a new question: https://ai.stackexchange.com/questions/46033/is-deep-learning-suitable-preferable-for-string-similarity-detection-and-applica  
  
  ---
  
  