Answers.Body: <p>I'm not sure I fully understand the use but I'll hazard at something real quick and see if it helps.</p>
  <p>If the agent has a perfect knowledge of environment dynamics then the policy can deal with states instead of actions.  The Agent in <span class="math-container">$S_1$</span>, knowing it has transitions available for it to <span class="math-container">$\{S_2, S_3, S_4\}$</span>, can determine it would like to transition to one of connecting states irrespective of the underlying action to get there.  This would be the <span class="math-container">$V_\pi(s)$</span> value iteration scheme.</p>
  <p>In many real world cases the dynamics of the environment are not known (or not known exactly) so the agent cannot directly pick it's next state.  But the actions are often known and so the agent can repeatedly sample from a list of actions in states to get a feeling of the quality of an action in a state (<span class="math-container">$Q_\pi(s,a)$</span>).  There is no need for every action to have an effect mind you.  If an action in a state is &quot;unsupported&quot; it can simply lead back to the same state.  The environment must map an action to a transition but the human doesn't need to define what they mean.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T17:58:43.767Z
Answers.Text: Thank you, this was a good answer, but I do have 1 question, if for example the current state is separated into the input and output state, and the output state remains inside the environment dynamics without being coded into the observation state, if ideally the value iteration scheme model has pefect knowledge of the environment dynamics, does that mean it knows to what state it should transition the input state to make the output state == goal state / max reward?
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-09T18:45:48.393Z
Answers.Text: Value iteration is the process by which we estimate the value of being in a state (based on the reward for getting there and discounted rewards of any subsequent states).  The policy which arises from value iteration is that for every state we choose the next state which has the maximum total reward.  Assuming our estimations of all states are near perfect, this policy can be followed from starting state to terminal in a path which gives max average reward.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T15:13:35.847Z
Answers.Text: So if ideally the true optimal values are known / perfect knowledge, even if the reward structure is sparse (recieves -1 for every state when goal isn't reached, +10 if goal state is reached), it will always know the path to the goal state in the policy value iteration function?
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-10T15:36:31.787Z
Answers.Text: Yes, so under value iteration the value of the state is the reward for reaching that state plus the discounted reward of next state(s).  Lets say the discount rate is .50.  If there is a -1 reward for the current state and 10 for the next, the value of the state is 4.  The state leading to this state would have a value of 1.  Compare this to a path which does not lead to a reward of 10 which would just be sums of discounted -1s and thus have lower value than the one that does.  It forms a sort of pheromone trail to a goal.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-11T16:06:58.163Z
Answers.Text: Oh I understand now, so basically if the policy value iteration function knows the true optimal values, it can find the optimal path since it already "knows" it.
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-11T21:56:51.353Z
Answers.Text: Hey uh actually can we get in touch? It's fine if you don't want to, I'm having some problems with my DQN model and I'm afraid if I post in AI StackExchange it's gonna get taken down for whatever reason, or i'm gonna get downvoted to oblivion
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-12T14:14:53.847Z
Answers.Text: Yea, AI StackExchange is for theory not programming.  I'd suggest posting it on StackOverflow with the correct tags.  I'm notoriously bad at reading other people's code so even if I can't help maybe someone else can.  Link it here I guess at the risk of it being marked as "discussion" but I should see it.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-12T22:12:03.433Z
Answers.Text: I was actually thinking about on a platform like Discord maybe? Yeah, I got a bad reputation there for apparently posting questions not related to topic
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-13T16:40:33.280Z
Answers.Text: Oh yeah and 1 question more, would the output layer of policy value iteration function in a Neural Network be continous or discrete? Would I have to set up individual states, or can it be a continous range of states?
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T15:22:41.890Z
Answers.Text: Hey you still there? I'm sorry if I went too far, I can settle in with StackOverflow if you're not comfortable with Discord
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T15:53:38.570Z
Answers.Text: I don’t know that I remember my discord login.  There is an RL community on there IIRC.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T16:02:00.630Z
Answers.Text: DQN + continuous output isn’t common.  The more common approach is a policy gradient method like Actor Critic, A3C, PPO, etc.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T19:13:19.917Z
Answers.Text: I mean like for the policy value iteration function, not a DQN
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T19:13:57.840Z
Answers.Text: And ok then, how about you create a new Discord account?
Answers.UserDisplayName: 19216811
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-08T21:59:38.397Z
Answers.LastActivityDate: 2024-04-08T21:59:38.397Z
Answers.UserDisplayName: foreverska
Body: <p>Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment? For example a function G(s) where it takes in the state as input and outputs an action based on it? This would help when the expert doesn't know how to define the actions right?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-08T01:41:56.997Z
Comments.Text: Are you asking if the network can from it’s environment determine a list of actions automatically and take one?
Comments.UserDisplayName: foreverska
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-08T16:23:44.657Z
Comments.Text: Sort of, like based on the environment it generates an action without needing the human to define any actions (eg. Moving right or left, velocity values, etc.)
Comments.UserDisplayName: 19216811
Tags: <neural-networks><reinforcement-learning><action-spaces>
Title: Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment?
UserDisplayName: 19216811
ThreadText: # Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment?
  
  (Asked by: username_0 on 2024-04-07)
  
  Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment? For example a function G(s) where it takes in the state as input and outputs an action based on it? This would help when the expert doesn't know how to define the actions right?
  
  **Tags:** neural-networks, reinforcement-learning, action-spaces
  
  > **Comments:**
  > **username_1**: Are you asking if the network can from it’s environment determine a list of actions automatically and take one?  
  > **username_0**: Sort of, like based on the environment it generates an action without needing the human to define any actions (eg. Moving right or left, velocity values, etc.)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-08. Score: 0 | Accepted answer
  
  I'm not sure I fully understand the use but I'll hazard at something real quick and see if it helps.
  
  If the agent has a perfect knowledge of environment dynamics then the policy can deal with states instead of actions. The Agent in $S\_1$, knowing it has transitions available for it to $\{S\_2, S\_3, S\_4\}$, can determine it would like to transition to one of connecting states irrespective of the underlying action to get there. This would be the $V\_\pi(s)$ value iteration scheme.
  
  In many real world cases the dynamics of the environment are not known (or not known exactly) so the agent cannot directly pick it's next state. But the actions are often known and so the agent can repeatedly sample from a list of actions in states to get a feeling of the quality of an action in a state ($Q\_\pi(s,a)$). There is no need for every action to have an effect mind you. If an action in a state is "unsupported" it can simply lead back to the same state. The environment must map an action to a transition but the human doesn't need to define what they mean.
  
  > **Comments:**
  > **username_0**: Thank you, this was a good answer, but I do have 1 question, if for example the current state is separated into the input and output state, and the output state remains inside the environment dynamics without being coded into the observation state, if ideally the value iteration scheme model has pefect knowledge of the environment dynamics, does that mean it knows to what state it should transition the input state to make the output state == goal state / max reward?  
  > **username_1**: Value iteration is the process by which we estimate the value of being in a state (based on the reward for getting there and discounted rewards of any subsequent states).  The policy which arises from value iteration is that for every state we choose the next state which has the maximum total reward.  Assuming our estimations of all states are near perfect, this policy can be followed from starting state to terminal in a path which gives max average reward.  
  > **username_0**: So if ideally the true optimal values are known / perfect knowledge, even if the reward structure is sparse (recieves -1 for every state when goal isn't reached, +10 if goal state is reached), it will always know the path to the goal state in the policy value iteration function?  
  > **username_1**: Yes, so under value iteration the value of the state is the reward for reaching that state plus the discounted reward of next state(s).  Lets say the discount rate is .50.  If there is a -1 reward for the current state and 10 for the next, the value of the state is 4.  The state leading to this state would have a value of 1.  Compare this to a path which does not lead to a reward of 10 which would just be sums of discounted -1s and thus have lower value than the one that does.  It forms a sort of pheromone trail to a goal.  
  > **username_0**: Oh I understand now, so basically if the policy value iteration function knows the true optimal values, it can find the optimal path since it already "knows" it.  
  > **username_0**: Hey uh actually can we get in touch? It's fine if you don't want to, I'm having some problems with my DQN model and I'm afraid if I post in AI StackExchange it's gonna get taken down for whatever reason, or i'm gonna get downvoted to oblivion  
  > **username_1**: Yea, AI StackExchange is for theory not programming.  I'd suggest posting it on StackOverflow with the correct tags.  I'm notoriously bad at reading other people's code so even if I can't help maybe someone else can.  Link it here I guess at the risk of it being marked as "discussion" but I should see it.  
  > **username_0**: I was actually thinking about on a platform like Discord maybe? Yeah, I got a bad reputation there for apparently posting questions not related to topic  
  > **username_0**: Oh yeah and 1 question more, would the output layer of policy value iteration function in a Neural Network be continous or discrete? Would I have to set up individual states, or can it be a continous range of states?  
  > **username_0**: Hey you still there? I'm sorry if I went too far, I can settle in with StackOverflow if you're not comfortable with Discord  
  > **username_1**: I don’t know that I remember my discord login.  There is an RL community on there IIRC.  
  > **username_1**: DQN + continuous output isn’t common.  The more common approach is a policy gradient method like Actor Critic, A3C, PPO, etc.  
  > **username_0**: I mean like for the policy value iteration function, not a DQN  
  > **username_0**: And ok then, how about you create a new Discord account?  
  
  ---
  
  