Options Discovery with Budgeted Reinforcement Learning
We consider the problem of learning hierarchical policies for Reinforcement Learning able to discover options, an option corresponding to a sub-policy over a set of primitive actions. Different models have been proposed during the last decade that usually rely on a predefined set of options. We specifically address the problem of automatically discovering options in decision processes. We describe a new RL learning framework called Bi-POMDP, and a new learning model called Budgeted Option Neural Network (BONN) able to discover options based on a budgeted learning objective. Since Bi-POMDP are more general than POMDP, our model can also be used to discover options for classical RL tasks. The BONN model is evaluated on different classical RL problems, demonstrating both quantitative and qualitative interesting results.
The paper tackles the very important problem of learning options from data. The introduction of the budget constraint is an interesting twist on this problem, which I had not seen before (though other methods apply other constraints.) I must say I’m not very convinced by the need to introduce the Bi-POMDP framework, where the conventional POMDP framework would do. In discussions, the authors suggest this makes for simpler comparison with RL models, but I find that it rather obscures the link to POMDP models. The proposed method makes an interesting contribution, distinct from the existing literature as far as I know. The extension to discover a discrete set of options is a nice feature for practical applications. In terms of the algorithm itself, I am actually unclear about lines 4 & 6. At line 4, I don’t know how sigma_t is computed. Can you give the precise equation? At line 6, I don’t know how the new option o_t is generated. Again, can you give the precise procedure? The paper contains several empirical results, on contrasting simulated domains. For some of these domains, such as CartPole, it’s really not clear that options are necessary. In my mind, the lack of comparison to other options learning methods is a limitation of the current draft.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Options Discovery with Budgeted Reinforcement Learning
We consider the problem of learning hierarchical policies for Reinforcement Learning able to discover options, an option corresponding to a sub-policy over a set of primitive actions. Different models have been proposed during the last decade that usually rely on a predefined set of options. We specifically address the problem of automatically discovering options in decision processes. We describe a new RL learning framework called Bi-POMDP, and a new learning model called Budgeted Option Neural Network (BONN) able to discover options based on a budgeted learning objective. Since Bi-POMDP are more general than POMDP, our model can also be used to discover options for classical RL tasks. The BONN model is evaluated on different classical RL problems, demonstrating both quantitative and qualitative interesting results.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper tackles the very important problem of learning options from data. The introduction of the budget constraint is an interesting twist on this problem, which I had not seen before (though other methods apply other constraints.) I must say I’m not very convinced by the need to introduce the Bi-POMDP framework, where the conventional POMDP framework would do. In discussions, the authors suggest this makes for simpler comparison with RL models, but I find that it rather obscures the link to POMDP models. The proposed method makes an interesting contribution, distinct from the existing literature as far as I know. The extension to discover a discrete set of options is a nice feature for practical applications. In terms of the algorithm itself, I am actually unclear about lines 4 & 6. At line 4, I don’t know how sigma_t is computed. Can you give the precise equation? At line 6, I don’t know how the new option o_t is generated. Again, can you give the precise procedure? The paper contains several empirical results, on contrasting simulated domains. For some of these domains, such as CartPole, it’s really not clear that options are necessary. In my mind, the lack of comparison to other options learning methods is a limitation of the current draft.
assistant
The paper tackles the very important problem of learning options from data. The introduction of the budget constraint is an interesting twist on this problem, which I had not seen before (though other methods apply other constraints.) I must say I’m not very convinced by the need to introduce the Bi-POMDP framework, where the conventional POMDP framework would do. In discussions, the authors suggest this makes for simpler comparison with RL models, but I find that it rather obscures the link to POMDP models. The proposed method makes an interesting contribution, distinct from the existing literature as far as I know. The extension to discover a discrete set of options is a nice feature for practical applications. In terms of the algorithm itself, I am actually unclear about lines 4 & 6. At line 4, I don’t know how sigma_t is computed. Can you give the precise equation? At line 6, I don’t know how the new option o_t is generated. Again, can you give the precise procedure? The paper contains several empirical results, on contrasting simulated domains. For some of these domains, such as CartPole, it’s really not clear that options are necessary. In my mind, the lack of comparison to other options learning methods is a limitation of the current draft.