Answers.Body: <p>It does not matter to the bandit algorithm that rewards are quantised or fractional, or that they can vary. This is true for pretty much all bandit optimisation algorithms.</p>
  
  <p>So just treat the $0.80 donation as a real valued reward of 0.8, that occurs on a single timestep.</p>
  
  <p>Treating a single reward on a single timestep as if it were multiple rewards across multiple timesteps might cause problems, depending on which algorithm you were using, even if the reward averaged to the same value. For instance it may skew the maths away from theoretical assumptions used in upper confidence bound action selection or gradient-based solutions.</p>
  
  <hr>
  
  <p>It occurs to me that your $1 experiment is supposed to be some kind of proxy for the real deployment, where donations will be sparse and may vary more. Whilst this may help a little, essentially you are just setting some kind of prior based on results from a test. The real bandit algorithm starts once the system is deployed into production, and could well have very different results when the goal is to collect real and larger donations.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-12T11:21:46.350Z
Answers.LastActivityDate: 2018-02-12T11:28:25.123Z
Answers.LastEditDate: 2018-02-12T11:28:25.123Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm developing a multi-armed bandit which learns the best information to display to persuade someone to donate to charity.</p>
  
  <p>Suppose I have treatments A, B, C, D (which are each one paragraph of text). The bandit selects one treatment to show to a person. The person is given $1 and has to decide how much (if any) to donate, in increments of one cent. The donation decision is recorded and fed to the multi-armed bandit, who will then re-optimize before another person is shown a treatment selected by the bandit.</p>
  
  <p>How should I program the bandit if my objective is to maximize total donations? For example, can I use Thompson sampling, and if a participant donates $0.80, I count that as 80 successes and 20 failures?</p>
  
Tags: <reinforcement-learning><multi-armed-bandits>
Title: Programming a bandit to optimize donations
UserDisplayName: wwl
ThreadText: # Programming a bandit to optimize donations
  
  (Asked by: username_0 on 2018-02-11)
  
  I'm developing a multi-armed bandit which learns the best information to display to persuade someone to donate to charity.
  
  Suppose I have treatments A, B, C, D (which are each one paragraph of text). The bandit selects one treatment to show to a person. The person is given $1 and has to decide how much (if any) to donate, in increments of one cent. The donation decision is recorded and fed to the multi-armed bandit, who will then re-optimize before another person is shown a treatment selected by the bandit.
  
  How should I program the bandit if my objective is to maximize total donations? For example, can I use Thompson sampling, and if a participant donates $0.80, I count that as 80 successes and 20 failures?
  
  **Tags:** reinforcement-learning, multi-armed-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-02-12. Score: 3 | Accepted answer
  
  It does not matter to the bandit algorithm that rewards are quantised or fractional, or that they can vary. This is true for pretty much all bandit optimisation algorithms.
  
  So just treat the $0.80 donation as a real valued reward of 0.8, that occurs on a single timestep.
  
  Treating a single reward on a single timestep as if it were multiple rewards across multiple timesteps might cause problems, depending on which algorithm you were using, even if the reward averaged to the same value. For instance it may skew the maths away from theoretical assumptions used in upper confidence bound action selection or gradient-based solutions.
  
  ---
  
  It occurs to me that your $1 experiment is supposed to be some kind of proxy for the real deployment, where donations will be sparse and may vary more. Whilst this may help a little, essentially you are just setting some kind of prior based on results from a test. The real bandit algorithm starts once the system is deployed into production, and could well have very different results when the goal is to collect real and larger donations.
  
  
  ---
  
  