<p>If I am understanding your question properly, you could use something like the following: </p>

<pre><code>import numpy as np 
p1 = 0.1
if np.random.rand() &lt; p1: 
    action = 'greedy'
else: 
    action = np.random.choice(['other_policy1', 'other_policy2', 'other_policy3']) 

return action 

</code></pre>

CC BY-SA 4.0
2019-07-19T10:12:03.117Z
Okay, what I have is like a list of actions say x = [1, 5, 6, 7, 8, 9, 10], and let us say the greatest element in the list has a probability p = 0.7 of being picked and all the rest have a probability p = 0.3 of being picked.
EArwa
CC BY-SA 4.0
2019-07-19T16:05:38.617Z
what do you mean by greatest?
It could be that you have a list of actions, and each action has a probability of being picked 
```
actions = [1,3,4]
np.random.choice(actions, p=[0.7, 0.15, 0.15])
```
If by greatest you mean the greatest value (in your example, 10), then you could use 
```
greatest_element = np.max(x)
if np.random.rand() < 0.7: 
    action = greatest_element
else: 
    action = np.random.choice(list(set(a) - {greatest_element}))
```
yahiaelgamal
CC BY-SA 4.0
2019-07-19T16:07:02.560Z
Like in the list, the greatest element is 10.
EArwa
CC BY-SA 4.0
2019-07-19T16:11:24.600Z
Okay, I see. But how do I enforce that probability if I do not know the values in the list but I want that greatest value to have a probability p1 of being picked and the rest probability p2 of being picked.
EArwa
CC BY-SA 4.0
2019-07-19T16:13:47.870Z
What does the np.random.rand() do? or you mean np.random.randn()?
EArwa
CC BY-SA 4.0
2019-07-19T16:14:50.620Z
https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.rand.html
generates a random number between 0-1
yahiaelgamal
CC BY-SA 4.0
2019-07-19T16:30:41.177Z
Okay, I see how it works. Let me test it with say 1000 random pickings.
EArwa
CC BY-SA 4.0
2019-07-19T16:42:47.773Z
import numpy as np

def picking():
    x = [1, 3, 5, 6, 8, 10]
    great = np.max(x)
    if np.random.rand() < 0.7:
        action = great
    else: 
        action = np.random.choice(list(set(x) - {great}))

    return(action)

N = 10000000
cg = 0
cr = 0

for i in range(N):
    y = picking()    
    if y == great:
        cg += 1
    else:
        cr += 1
print(cg/N)
print(cr/N)
EArwa
CC BY-SA 4.0
2019-07-19T16:45:06.767Z
It is correct but it requires a very large number of pickings to converge to the enforced probabilities. Even with 10million pickings the results still are 0.7001996 and  0.2998004.
EArwa
CC BY-SA 4.0
2019-07-19T16:52:16.913Z
That's how probabilities work. If you "enforce" by making exactly 7 in 10 be the greatest element. Depending on how many you have passed, it will never be exactly 0.7. (e.g. if you run 10 times, and you "enforce" it, it will be 0.7, if you run it 11, it will 0.63). You will never have exactly 0.7. Even if you enforce it with a non-random choice making (e.g. i%10 <= 7). It will won't be exactly 0.7. And it's not random at all
yahiaelgamal
CC BY-SA 4.0
2019-07-21T09:25:10.130Z
I have tried random.choice((great)*70 + (list(set(x) - {great})*30) and it converges to exactly 0.7 and 0.3 in less than 50 000 pickings.
EArwa
CC BY-SA 4.0
2019-07-21T17:45:03.230Z
I highly recommended reviewing basic probability theory in order to avoid being deceived by randomness.
yahiaelgamal
CC BY-SA 4.0
2019-07-19T09:56:52.770Z
2019-07-19T09:56:52.770Z
yahiaelgamal
<p>In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say <span class="math-container">$p_1$</span> (updated every episode) of being selected, while the rest have a probability <span class="math-container">$p_2$</span> (updated every episode) of being selected. </p>

<p>Could you please show me an example code (Python) on how to enforce such conditional probabilistic picking?</p>

CC BY-SA 4.0
2019-07-19T09:04:14.150Z
[numpy.random.choice](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.random.choice.html) lets you do weighted random choice, study the docs for more info.
Brale
CC BY-SA 4.0
2019-07-19T10:14:57.303Z
numpy.random.choice expects a probability distribution such that if the probability of picking the elements are added individually, it would sum to 1. But in this one, a particular value in the list has a probability say p =0.7 while all the rest have p = 0.3. If you try numpy.random.choice, it returns an error since the distribution is greater than 1.
EArwa
CC BY-SA 4.0
2019-07-19T10:20:21.377Z
action probabilities need to sum up to 1. You cannot have probability higher than 1. You misunderstood the part about picking other actions. If greedy action is picked with probability of 0.7 then other actions **together** have probability 0.3 of being picked, meaning each other action has probability of $\frac{0.3}{n-1}$ of being picked.
Brale
CC BY-SA 4.0
2019-07-19T10:21:59.493Z
also in pursuit algorithm its not guaranteed that other non greedy actions have same probability of being picked, probabilities are updated each on their own, but all action probabilities always need to sum up to 1
Brale
CC BY-SA 4.0
2019-07-19T12:53:38.100Z
Okay, so assuming I did say N pickings (for N sufficiently large), and say the greedy action is picked n1 times while the rest picked n2 times; will I get n1/N = 0.7 and n2/N = 0.3?
EArwa
CC BY-SA 4.0
2019-07-19T14:17:23.853Z
if by "the rest picked" you mean all those other actions together in total were picked $n_2$ times then yes, that's correct.
Brale
CC BY-SA 4.0
2019-07-19T16:05:48.470Z
Let me see how that works. I was trying something like: '
    action = random.choice((greedy_action)*8 + (not_greedy)*2)
EArwa
<reinforcement-learning><q-learning>
Probabilistic action selection in pursuit algorithm
EArwa
# Probabilistic action selection in pursuit algorithm

(Asked by: username_0 on 2019-07-19)

In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p\_1$ (updated every episode) of being selected, while the rest have a probability $p\_2$ (updated every episode) of being selected.

Could you please show me an example code (Python) on how to enforce such conditional probabilistic picking?

**Tags:** reinforcement-learning, q-learning

> **Comments:**
> **username_1**: [numpy.random.choice](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.random.choice.html) lets you do weighted random choice, study the docs for more info.  
> **username_0**: numpy.random.choice expects a probability distribution such that if the probability of picking the elements are added individually, it would sum to 1. But in this one, a particular value in the list has a probability say p =0.7 while all the rest have p = 0.3. If you try numpy.random.choice, it returns an error since the distribution is greater than 1.  
> **username_1**: action probabilities need to sum up to 1. You cannot have probability higher than 1. You misunderstood the part about picking other actions. If greedy action is picked with probability of 0.7 then other actions **together** have probability 0.3 of being picked, meaning each other action has probability of $\frac{0.3}{n-1}$ of being picked.  
> **username_1**: also in pursuit algorithm its not guaranteed that other non greedy actions have same probability of being picked, probabilities are updated each on their own, but all action probabilities always need to sum up to 1  
> **username_0**: Okay, so assuming I did say N pickings (for N sufficiently large), and say the greedy action is picked n1 times while the rest picked n2 times; will I get n1/N = 0.7 and n2/N = 0.3?  
> **username_1**: if by "the rest picked" you mean all those other actions together in total were picked $n_2$ times then yes, that's correct.  
> **username_0**: Let me see how that works. I was trying something like: '     action = random.choice((greedy_action)*8 + (not_greedy)*2)  

---

## 1 answers

### Answer by username_2 on 2019-07-19. Score: 1 | Accepted answer

If I am understanding your question properly, you could use something like the following: 
import numpy as np 
p1 = 0.1
if np.random.rand() < p1: 
    action = 'greedy'
else: 
    action = np.random.choice(['other_policy1', 'other_policy2', 'other_policy3']) 

return action

> **Comments:**
> **username_0**: Okay, what I have is like a list of actions say x = [1, 5, 6, 7, 8, 9, 10], and let us say the greatest element in the list has a probability p = 0.7 of being picked and all the rest have a probability p = 0.3 of being picked.  
> **username_2**: what do you mean by greatest? It could be that you have a list of actions, and each action has a probability of being picked  ``` actions = [1,3,4] np.random.choice(actions, p=[0.7, 0.15, 0.15]) ``` If by greatest you mean the greatest value (in your example, 10), then you could use  ``` greatest_element = np.max(x) if np.random.rand() < 0.7:      action = greatest_element else:      action = np.random.choice(list(set(a) - {greatest_element})) ```  
> **username_0**: Like in the list, the greatest element is 10.  
> **username_0**: Okay, I see. But how do I enforce that probability if I do not know the values in the list but I want that greatest value to have a probability p1 of being picked and the rest probability p2 of being picked.  
> **username_0**: What does the np.random.rand() do? or you mean np.random.randn()?  
> **username_2**: https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.rand.html generates a random number between 0-1  
> **username_0**: Okay, I see how it works. Let me test it with say 1000 random pickings.  
> **username_0**: import numpy as np  def picking():     x = [1, 3, 5, 6, 8, 10]     great = np.max(x)     if np.random.rand() < 0.7:         action = great     else:          action = np.random.choice(list(set(x) - {great}))      return(action)  N = 10000000 cg = 0 cr = 0  for i in range(N):     y = picking()         if y == great:         cg += 1     else:         cr += 1 print(cg/N) print(cr/N)  
> **username_0**: It is correct but it requires a very large number of pickings to converge to the enforced probabilities. Even with 10million pickings the results still are 0.7001996 and  0.2998004.  
> **username_2**: That's how probabilities work. If you "enforce" by making exactly 7 in 10 be the greatest element. Depending on how many you have passed, it will never be exactly 0.7. (e.g. if you run 10 times, and you "enforce" it, it will be 0.7, if you run it 11, it will 0.63). You will never have exactly 0.7. Even if you enforce it with a non-random choice making (e.g. i%10 <= 7). It will won't be exactly 0.7. And it's not random at all  
> **username_0**: I have tried random.choice((great)*70 + (list(set(x) - {great})*30) and it converges to exactly 0.7 and 0.3 in less than 50 000 pickings.  
> **username_2**: I highly recommended reviewing basic probability theory in order to avoid being deceived by randomness.  

---

