Answers.Body: <p>If I am understanding your question properly, you could use something like the following: </p>
  
  <pre><code>import numpy as np 
  p1 = 0.1
  if np.random.rand() &lt; p1: 
      action = 'greedy'
  else: 
      action = np.random.choice(['other_policy1', 'other_policy2', 'other_policy3']) 
  
  return action 
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T10:12:03.117Z
Answers.Text: Okay, what I have is like a list of actions say x = [1, 5, 6, 7, 8, 9, 10], and let us say the greatest element in the list has a probability p = 0.7 of being picked and all the rest have a probability p = 0.3 of being picked.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:05:38.617Z
Answers.Text: what do you mean by greatest?
  It could be that you have a list of actions, and each action has a probability of being picked 
  ```
  actions = [1,3,4]
  np.random.choice(actions, p=[0.7, 0.15, 0.15])
  ```
  If by greatest you mean the greatest value (in your example, 10), then you could use 
  ```
  greatest_element = np.max(x)
  if np.random.rand() < 0.7: 
      action = greatest_element
  else: 
      action = np.random.choice(list(set(a) - {greatest_element}))
  ```
Answers.UserDisplayName: yahiaelgamal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:07:02.560Z
Answers.Text: Like in the list, the greatest element is 10.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:11:24.600Z
Answers.Text: Okay, I see. But how do I enforce that probability if I do not know the values in the list but I want that greatest value to have a probability p1 of being picked and the rest probability p2 of being picked.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:13:47.870Z
Answers.Text: What does the np.random.rand() do? or you mean np.random.randn()?
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:14:50.620Z
Answers.Text: https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.rand.html
  generates a random number between 0-1
Answers.UserDisplayName: yahiaelgamal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:30:41.177Z
Answers.Text: Okay, I see how it works. Let me test it with say 1000 random pickings.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:42:47.773Z
Answers.Text: import numpy as np
  
  def picking():
      x = [1, 3, 5, 6, 8, 10]
      great = np.max(x)
      if np.random.rand() < 0.7:
          action = great
      else: 
          action = np.random.choice(list(set(x) - {great}))
  
      return(action)
  
  N = 10000000
  cg = 0
  cr = 0
  
  for i in range(N):
      y = picking()    
      if y == great:
          cg += 1
      else:
          cr += 1
  print(cg/N)
  print(cr/N)
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:45:06.767Z
Answers.Text: It is correct but it requires a very large number of pickings to converge to the enforced probabilities. Even with 10million pickings the results still are 0.7001996 and  0.2998004.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T16:52:16.913Z
Answers.Text: That's how probabilities work. If you "enforce" by making exactly 7 in 10 be the greatest element. Depending on how many you have passed, it will never be exactly 0.7. (e.g. if you run 10 times, and you "enforce" it, it will be 0.7, if you run it 11, it will 0.63). You will never have exactly 0.7. Even if you enforce it with a non-random choice making (e.g. i%10 <= 7). It will won't be exactly 0.7. And it's not random at all
Answers.UserDisplayName: yahiaelgamal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-21T09:25:10.130Z
Answers.Text: I have tried random.choice((great)*70 + (list(set(x) - {great})*30) and it converges to exactly 0.7 and 0.3 in less than 50 000 pickings.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-21T17:45:03.230Z
Answers.Text: I highly recommended reviewing basic probability theory in order to avoid being deceived by randomness.
Answers.UserDisplayName: yahiaelgamal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-19T09:56:52.770Z
Answers.LastActivityDate: 2019-07-19T09:56:52.770Z
Answers.UserDisplayName: yahiaelgamal
Body: <p>In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say <span class="math-container">$p_1$</span> (updated every episode) of being selected, while the rest have a probability <span class="math-container">$p_2$</span> (updated every episode) of being selected. </p>
  
  <p>Could you please show me an example code (Python) on how to enforce such conditional probabilistic picking?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T09:04:14.150Z
Comments.Text: [numpy.random.choice](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.random.choice.html) lets you do weighted random choice, study the docs for more info.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T10:14:57.303Z
Comments.Text: numpy.random.choice expects a probability distribution such that if the probability of picking the elements are added individually, it would sum to 1. But in this one, a particular value in the list has a probability say p =0.7 while all the rest have p = 0.3. If you try numpy.random.choice, it returns an error since the distribution is greater than 1.
Comments.UserDisplayName: EArwa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T10:20:21.377Z
Comments.Text: action probabilities need to sum up to 1. You cannot have probability higher than 1. You misunderstood the part about picking other actions. If greedy action is picked with probability of 0.7 then other actions **together** have probability 0.3 of being picked, meaning each other action has probability of $\frac{0.3}{n-1}$ of being picked.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T10:21:59.493Z
Comments.Text: also in pursuit algorithm its not guaranteed that other non greedy actions have same probability of being picked, probabilities are updated each on their own, but all action probabilities always need to sum up to 1
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T12:53:38.100Z
Comments.Text: Okay, so assuming I did say N pickings (for N sufficiently large), and say the greedy action is picked n1 times while the rest picked n2 times; will I get n1/N = 0.7 and n2/N = 0.3?
Comments.UserDisplayName: EArwa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T14:17:23.853Z
Comments.Text: if by "the rest picked" you mean all those other actions together in total were picked $n_2$ times then yes, that's correct.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-19T16:05:48.470Z
Comments.Text: Let me see how that works. I was trying something like: '
      action = random.choice((greedy_action)*8 + (not_greedy)*2)
Comments.UserDisplayName: EArwa
Tags: <reinforcement-learning><q-learning>
Title: Probabilistic action selection in pursuit algorithm
UserDisplayName: EArwa
ThreadText: # Probabilistic action selection in pursuit algorithm
  
  (Asked by: username_0 on 2019-07-19)
  
  In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p\_1$ (updated every episode) of being selected, while the rest have a probability $p\_2$ (updated every episode) of being selected.
  
  Could you please show me an example code (Python) on how to enforce such conditional probabilistic picking?
  
  **Tags:** reinforcement-learning, q-learning
  
  > **Comments:**
  > **username_1**: [numpy.random.choice](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.random.choice.html) lets you do weighted random choice, study the docs for more info.  
  > **username_0**: numpy.random.choice expects a probability distribution such that if the probability of picking the elements are added individually, it would sum to 1. But in this one, a particular value in the list has a probability say p =0.7 while all the rest have p = 0.3. If you try numpy.random.choice, it returns an error since the distribution is greater than 1.  
  > **username_1**: action probabilities need to sum up to 1. You cannot have probability higher than 1. You misunderstood the part about picking other actions. If greedy action is picked with probability of 0.7 then other actions **together** have probability 0.3 of being picked, meaning each other action has probability of $\frac{0.3}{n-1}$ of being picked.  
  > **username_1**: also in pursuit algorithm its not guaranteed that other non greedy actions have same probability of being picked, probabilities are updated each on their own, but all action probabilities always need to sum up to 1  
  > **username_0**: Okay, so assuming I did say N pickings (for N sufficiently large), and say the greedy action is picked n1 times while the rest picked n2 times; will I get n1/N = 0.7 and n2/N = 0.3?  
  > **username_1**: if by "the rest picked" you mean all those other actions together in total were picked $n_2$ times then yes, that's correct.  
  > **username_0**: Let me see how that works. I was trying something like: '     action = random.choice((greedy_action)*8 + (not_greedy)*2)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-07-19. Score: 1 | Accepted answer
  
  If I am understanding your question properly, you could use something like the following:
  
  ```
  import numpy as np 
  p1 = 0.1
  if np.random.rand() < p1: 
      action = 'greedy'
  else: 
      action = np.random.choice(['other_policy1', 'other_policy2', 'other_policy3']) 
  
  return action 
  
  ```
  
  > **Comments:**
  > **username_0**: Okay, what I have is like a list of actions say x = [1, 5, 6, 7, 8, 9, 10], and let us say the greatest element in the list has a probability p = 0.7 of being picked and all the rest have a probability p = 0.3 of being picked.  
  > **username_2**: what do you mean by greatest? It could be that you have a list of actions, and each action has a probability of being picked  ``` actions = [1,3,4] np.random.choice(actions, p=[0.7, 0.15, 0.15]) ``` If by greatest you mean the greatest value (in your example, 10), then you could use  ``` greatest_element = np.max(x) if np.random.rand() < 0.7:      action = greatest_element else:      action = np.random.choice(list(set(a) - {greatest_element})) ```  
  > **username_0**: Like in the list, the greatest element is 10.  
  > **username_0**: Okay, I see. But how do I enforce that probability if I do not know the values in the list but I want that greatest value to have a probability p1 of being picked and the rest probability p2 of being picked.  
  > **username_0**: What does the np.random.rand() do? or you mean np.random.randn()?  
  > **username_2**: https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.rand.html generates a random number between 0-1  
  > **username_0**: Okay, I see how it works. Let me test it with say 1000 random pickings.  
  > **username_0**: import numpy as np  def picking():     x = [1, 3, 5, 6, 8, 10]     great = np.max(x)     if np.random.rand() < 0.7:         action = great     else:          action = np.random.choice(list(set(x) - {great}))      return(action)  N = 10000000 cg = 0 cr = 0  for i in range(N):     y = picking()         if y == great:         cg += 1     else:         cr += 1 print(cg/N) print(cr/N)  
  > **username_0**: It is correct but it requires a very large number of pickings to converge to the enforced probabilities. Even with 10million pickings the results still are 0.7001996 and  0.2998004.  
  > **username_2**: That's how probabilities work. If you "enforce" by making exactly 7 in 10 be the greatest element. Depending on how many you have passed, it will never be exactly 0.7. (e.g. if you run 10 times, and you "enforce" it, it will be 0.7, if you run it 11, it will 0.63). You will never have exactly 0.7. Even if you enforce it with a non-random choice making (e.g. i%10 <= 7). It will won't be exactly 0.7. And it's not random at all  
  > **username_0**: I have tried random.choice((great)*70 + (list(set(x) - {great})*30) and it converges to exactly 0.7 and 0.3 in less than 50 000 pickings.  
  > **username_2**: I highly recommended reviewing basic probability theory in order to avoid being deceived by randomness.  
  
  ---
  
  