Answers.Body: <p>If you want to treat the problem as a full Reinforcement Learning problem, I'd recommend to try avoiding the combinatorial explosion of the action space by treating every sub-action as a separate decision point, a separate full action. If you have, for example, already selected 4 sub-actions for a particular customer, you can try to include those in some way in the state representation / input when moving on to the 5th sub-action. By including already-selected sub-actions in the state space, your algorithm can learn to take into account that optimal prices for some seat types will depend on what prices were already selected for others.</p>
  
  <p>I do suspect such a full RL formulation will still be a difficult problem to learn though, will require huge amounts of experience. It may be worth considering to simplify it anyway, treat it as a Contextual + Combinatorial Multi-Armed Bandit Problem. That way you won't be able to learn long-term effects across multiple different customers as you described in the comments, but you will likely at least be able to learn something that works decently well with less experience. Recently, an interesting new book appeared on MAB problems, which is available for free <a href="http://downloads.tor-lattimore.com/banditbook/book.pdf" rel="nofollow noreferrer">here</a>. You will find many chapters on Contextual MABs there, and also one chapter (chapter 30) on Combinatorial MABs.</p>
  
  <p>Note that with either of these "combinatorial" approaches, you can also try to play around with the order in which you select sub-actions. For example, the sub-action / price point selected for seat type A might have a significant influence on what the optimal remaining policy would be for other seat types, whereas the sub-action for seat type B might have no influence on other seat types. It would then be useful to always prioritize selecting sub-actions for seat type A. You can try to identify these kinds of effects by keeping track of (co)variances in observed returns.</p>
  
  <hr>
  
  <blockquote>
    <ol>
    <li>Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?</li>
    </ol>
  </blockquote>
  
  <p>The solutions I proposed above do not do this explicitly, they circumvent the issue by taking multiple sequential decisions. There does appear to be some research in RL with vector-valued actions. For example, the paper <a href="https://arxiv.org/pdf/1802.07564" rel="nofollow noreferrer">Clipped Action Policy Gradient</a> briefly mentions vector-valued actions in Subsection 3.2. I am not personally familiar enough with RL + vector-valued actions to make a direct recommendation as to what approaches do or don't work well, but maybe this can at least help you find more relevant literature if this is a direction you'd like to pursue.</p>
  
  <blockquote>
    <ol start="2">
    <li>Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?</li>
    <li>Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?</li>
    </ol>
  </blockquote>
  
  <p>This kind of generalization should again come naturally from using function approximation with either of the solutions proposed above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-12T20:57:45.383Z
Answers.Text: Hi Dennis thanks a lot for your answer. I don't think I was clear about the prices, hopefully my comment in reply to your earlier one on the question clears up what I mean about the combinations. I've also edited the question a bit since I don't think I properly explained that I am looking to change prices dynamically over the selling cycle of the flight (and hence why I think a full MDP makes sense). I did consider a bandit type formulation, and indeed I think that would work best if there were not inventory constraints.
Answers.UserDisplayName: domdomdom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-12T21:03:47.290Z
Answers.Text: However, the action of selling seat reservations does affect the future state, since it reduces seat inventory that can be sold/reserved in future booking events. For example, I might be able to sell exit row reservations for a higher price later in the booking process since business customers start booking. Because of this I might decide to price them high early in the window, to discourage demand at a price lower than I believe I can get later. This may not give me the best expected reward from this one action, but may end up being optimal over the life of the flight.
Answers.UserDisplayName: domdomdom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T08:43:48.970Z
Answers.Text: @domdomdom Ah yes you're right, it seems like I was sleeping. Just edited the answer.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T09:20:38.493Z
Answers.Text: Thanks for the edit Dennis, it's been very useful. Was the answer supposed to contain a link to that book about MABs? If so it doesn't seem to work, would be great if you could link it in the comments, thanks!
Answers.UserDisplayName: domdomdom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-13T09:50:39.097Z
Answers.Text: @domdomdom Yes, the link was broken because it didn't start with `http://`, fixed now. I'll try to find time later today to edit in a bit more of a complete answer, addressing again the specific questions you asked about generalization.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T09:49:04.603Z
Answers.Text: The idea of using a bandit formulation is especially interesting, thanks for your help.
Answers.UserDisplayName: domdomdom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-12T18:12:13.937Z
Answers.LastActivityDate: 2018-08-13T11:44:10.180Z
Answers.LastEditDate: 2018-08-13T11:44:10.180Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I am trying to build an RL agent to price paid-for-seating on commercial flights. I should reiterate here - I am not talking about the price of the ticket - rather, I am talking about the pricing you see if you click on the seat map to choose where on the plane you sit (exits rows, window seats, etc). The general set up is:</p>
  
  <ol>
  <li>After choosing their flights (for a booking of <em>n</em> people), a customer will view a web page with the available seat types and their prices visible.</li>
  <li>They select between zero and <em>n</em> seats from a seat map with a variety of different prices for different seats, to be added to their booking.</li>
  <li>The revenue from step 2 is observed as the reward.</li>
  </ol>
  
  <p>Each 'episode' is the selling cycle of one flight. Whether the customer buys a chosen seat or not, the inventory goes down as they still have a ticket for the flight so will get a seat at departure. I would like to change prices on the fly, rather than fix a set of optimal prices throughout the selling cycle. </p>
  
  <p>I have not decided on a general architecture yet. I want to take various booking, flight, and inventory information into account, so I know I will be using function approximation (most likely a neural net) to generalise over the state space.</p>
  
  <p>However, I am less clear on how to set up my action space. I imagine an action would amount to a vector with a price for each different seat type (window seat, exit row, etc). If I have, for example, 8 different seat types, and 10 different price points for each, this gives me a total of 10^8 different actions, many of which will be very similar. In a sense, each action is comprised of a combination of sub-actions - the action of pricing each seat type.</p>
  
  <p>Additionally, each sub-action (pricing one seat type) is somewhat dependent on the others, in the sense that the price of one seat type will likely affect the demand (and hence reward contribution) for another. For example, if you set window seats to a very cheap price, people will be less likely to spend a normal amount for the other seat types. Hence, I doubt the problem can be decomposed into a set of sub-problems.</p>
  
  <p>I'm interested if there has been any research into dealing with a problem like this. Clearly any agent I build needs some way to generalise across actions to some degree, since collecting real data on millions of actions is not possible, even just for one state.</p>
  
  <p>As I see it, this comes down to three questions:</p>
  
  <ol>
  <li>Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?</li>
  <li>Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?</li>
  <li>Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?</li>
  </ol>
  
  <p>I haven't been able to find any literature on this, especially relating to the second question - any help would be much appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-12T22:03:27.917Z
Comments.Text: I may not have been clear about the seat price combinations. I need to set 8 different prices, each which have 10 different price points they could take, making 10^8 combinations. To take a smaller example, if we have seat types A, B and C, that can take value 10 or 15, we get combinations: 10, 10, 10 | 10, 10, 15 | 10, 15, 10 | 10, 15, 15 | 15, 10, 10 | 15, 15, 10 | 15, 10, 15 | 15, 15, 15. This is eight combinations, 2^3 - the number of price points to the power of the number of seat types. I hope this clears up the confusion. Thanks.
Comments.UserDisplayName: domdomdom
Tags: <reinforcement-learning><combinatorics>
Title: How to generalise over multiple simultaneous dependent actions in Reinforcement Learning
UserDisplayName: domdomdom
ThreadText: # How to generalise over multiple simultaneous dependent actions in Reinforcement Learning
  
  (Asked by: username_0 on 2018-08-12)
  
  I am trying to build an RL agent to price paid-for-seating on commercial flights. I should reiterate here - I am not talking about the price of the ticket - rather, I am talking about the pricing you see if you click on the seat map to choose where on the plane you sit (exits rows, window seats, etc). The general set up is:
  
  1. After choosing their flights (for a booking of *n* people), a customer will view a web page with the available seat types and their prices visible.
  2. They select between zero and *n* seats from a seat map with a variety of different prices for different seats, to be added to their booking.
  3. The revenue from step 2 is observed as the reward.
  
  Each 'episode' is the selling cycle of one flight. Whether the customer buys a chosen seat or not, the inventory goes down as they still have a ticket for the flight so will get a seat at departure. I would like to change prices on the fly, rather than fix a set of optimal prices throughout the selling cycle.
  
  I have not decided on a general architecture yet. I want to take various booking, flight, and inventory information into account, so I know I will be using function approximation (most likely a neural net) to generalise over the state space.
  
  However, I am less clear on how to set up my action space. I imagine an action would amount to a vector with a price for each different seat type (window seat, exit row, etc). If I have, for example, 8 different seat types, and 10 different price points for each, this gives me a total of 10^8 different actions, many of which will be very similar. In a sense, each action is comprised of a combination of sub-actions - the action of pricing each seat type.
  
  Additionally, each sub-action (pricing one seat type) is somewhat dependent on the others, in the sense that the price of one seat type will likely affect the demand (and hence reward contribution) for another. For example, if you set window seats to a very cheap price, people will be less likely to spend a normal amount for the other seat types. Hence, I doubt the problem can be decomposed into a set of sub-problems.
  
  I'm interested if there has been any research into dealing with a problem like this. Clearly any agent I build needs some way to generalise across actions to some degree, since collecting real data on millions of actions is not possible, even just for one state.
  
  As I see it, this comes down to three questions:
  
  1. Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?
  2. Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
  3. Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?
  
  I haven't been able to find any literature on this, especially relating to the second question - any help would be much appreciated!
  
  **Tags:** reinforcement-learning, combinatorics
  
  > **Comments:**
  > **username_0**: I may not have been clear about the seat price combinations. I need to set 8 different prices, each which have 10 different price points they could take, making 10^8 combinations. To take a smaller example, if we have seat types A, B and C, that can take value 10 or 15, we get combinations: 10, 10, 10 | 10, 10, 15 | 10, 15, 10 | 10, 15, 15 | 15, 10, 10 | 15, 15, 10 | 15, 10, 15 | 15, 15, 15. This is eight combinations, 2^3 - the number of price points to the power of the number of seat types. I hope this clears up the confusion. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-12. Score: 3 | Accepted answer
  
  If you want to treat the problem as a full Reinforcement Learning problem, I'd recommend to try avoiding the combinatorial explosion of the action space by treating every sub-action as a separate decision point, a separate full action. If you have, for example, already selected 4 sub-actions for a particular customer, you can try to include those in some way in the state representation / input when moving on to the 5th sub-action. By including already-selected sub-actions in the state space, your algorithm can learn to take into account that optimal prices for some seat types will depend on what prices were already selected for others.
  
  I do suspect such a full RL formulation will still be a difficult problem to learn though, will require huge amounts of experience. It may be worth considering to simplify it anyway, treat it as a Contextual + Combinatorial Multi-Armed Bandit Problem. That way you won't be able to learn long-term effects across multiple different customers as you described in the comments, but you will likely at least be able to learn something that works decently well with less experience. Recently, an interesting new book appeared on MAB problems, which is available for free [here](http://downloads.tor-lattimore.com/banditbook/book.pdf). You will find many chapters on Contextual MABs there, and also one chapter (chapter 30) on Combinatorial MABs.
  
  Note that with either of these "combinatorial" approaches, you can also try to play around with the order in which you select sub-actions. For example, the sub-action / price point selected for seat type A might have a significant influence on what the optimal remaining policy would be for other seat types, whereas the sub-action for seat type B might have no influence on other seat types. It would then be useful to always prioritize selecting sub-actions for seat type A. You can try to identify these kinds of effects by keeping track of (co)variances in observed returns.
  
  ---
  
  > 1. Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?
  
  The solutions I proposed above do not do this explicitly, they circumvent the issue by taking multiple sequential decisions. There does appear to be some research in RL with vector-valued actions. For example, the paper [Clipped Action Policy Gradient](https://arxiv.org/pdf/1802.07564) briefly mentions vector-valued actions in Subsection 3.2. I am not personally familiar enough with RL + vector-valued actions to make a direct recommendation as to what approaches do or don't work well, but maybe this can at least help you find more relevant literature if this is a direction you'd like to pursue.
  
  > 2. Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
  > 3. Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?
  
  This kind of generalization should again come naturally from using function approximation with either of the solutions proposed above.
  
  > **Comments:**
  > **username_0**: Hi username_1 thanks a lot for your answer. I don't think I was clear about the prices, hopefully my comment in reply to your earlier one on the question clears up what I mean about the combinations. I've also edited the question a bit since I don't think I properly explained that I am looking to change prices dynamically over the selling cycle of the flight (and hence why I think a full MDP makes sense). I did consider a bandit type formulation, and indeed I think that would work best if there were not inventory constraints.  
  > **username_0**: However, the action of selling seat reservations does affect the future state, since it reduces seat inventory that can be sold/reserved in future booking events. For example, I might be able to sell exit row reservations for a higher price later in the booking process since business customers start booking. Because of this I might decide to price them high early in the window, to discourage demand at a price lower than I believe I can get later. This may not give me the best expected reward from this one action, but may end up being optimal over the life of the flight.  
  > **username_1**: @username_0 Ah yes you're right, it seems like I was sleeping. Just edited the answer.  
  > **username_0**: Thanks for the edit username_1, it's been very useful. Was the answer supposed to contain a link to that book about MABs? If so it doesn't seem to work, would be great if you could link it in the comments, thanks!  
  > **username_1**: @username_0 Yes, the link was broken because it didn't start with `http://`, fixed now. I'll try to find time later today to edit in a bit more of a complete answer, addressing again the specific questions you asked about generalization.  
  > **username_0**: The idea of using a bandit formulation is especially interesting, thanks for your help.  
  
  ---
  
  