Optimizing Energy Production Using Policy Search and Predictive State Representations
We consider the challenging practical problem of optimizing the power production of a complex of hydroelectric power plants, which involves control over three continuous action variables, uncertainty in the amount of water inflows and a variety of constraints that need to be satisfied. We propose a policy-search-based approach coupled with predictive modelling to address this problem. This approach has some key advantages compared to other alternatives, such as dynamic programming: the policy representation and search algorithm can conveniently incorporate domain knowledge; the resulting policies are easy to interpret, and the algorithm is naturally parallelizable. Our algorithm obtains a policy which outperforms the solution found by dynamic programming both quantitatively and qualitatively.
This is an application paper that attacks the problem of managingturbine speeds of four dams in Quebec. The goal is to maximize energyproduction (equivalent to maximizing turbine speeds) subject tovarious ecological and operational constraints. Interestingly, a full dynamical model for this problem isavailable. However, this is a continuous state, continuous action,discrete time (weekly-time-step) problem, so applying standard DPmethods to solve the problem requires discretization. This in turnleads to non-smooth policies (i.e., turbine rates jump around becauseof discretization), that are hard to explain (and undesirable fornon-represented operational reasons) to the operators.Instead of DP, this paper pursues a PSR approach in which theexisting simulator (a custom auto-regressive model) and historicaldata are combined to create a dynamical model. One relevant statevariable (month of year) is observable, so the authors incorporatethis variable into the PSR via the methods of [8]. The learn PSRswith and without water input forecasts. Constraints are incorporated by a mix of penalty terms, fixing thepolicy in certain time periods, and partitioning the state space intotwo time periods and learning a separate policy for each time period. The results evaluate performance on many dimensions including (a)total energy produced, (b) frequency with which constraints areviolated, and (c) severity with which constraints are violated.(a) and (c) are improved but at a slight cost of (b). In addition, thepolicies are qualitatively much nicer. Quality: The paper is well written and the material is reasonablyself-contained, with appropriate references to prior work.  The newapproach is compared to a state-of-the-art solution developed by theQuebec water management organization, which is exactly the rightbaseline. The fact that they are able to significantly improve uponthe DP policy is very interesting and has wider lessons for all of uswho work on applied reinforcement learning methods.  In particular,the need for explanable/visualizable policies is not sufficientlyappreciated in the research community. This paper helps address thisproblem.  This is a very nice example of applying a mix of advancedtechniques to show an important application improvement.One question: The authors say that it is difficult (or impossible) toincorporate the steady-flow constraint for weeks 43-45 into a DPmethod. But couldn't that be done within an SMDP framework, where theaction for weeks 43-45 is treated as a single action with a 3-weekduration?  This is more or less what the authors did in theirhand-coded policy for R2. Clarity: The paper is well-written with few exceptions.    Line 142: "or" -> "of"?             insert "the" before "logarithmic case"?   Line 209: You introduce T without defining it. You need to remind   the reader what a "test" is.    Line 236: Should "assuming" be "assume"? You don't discuss Figure 2   very much. Is there more to say?Originality: The work is original within the RL literature. I don't   know about the reservoir management literature. Significance: wrt the research community, the paper is significantbecause it shows (a) the virtues of PSRs for achieving generality, (b)the advantages of policy search for incorporating domain constraintsand achieving smoother policies, and (c) the importance ofinterpretability.  In the reservoir management literature and thelarger sustainability literature, this work is important because itdemonstrates that there is significant advantage in applying thesophisticated techniques coming out of the research community. Thispaper also points the way toward many important areas for futurework. For example, under climate change, the variance in precipitation(and hence, reservoir inputs) can be expected to increase.  Thetechniques in this paper could easily be applied to model the impactof that increased variance on reservoir management and may suggest aneed for additional reservoir capacity and other management changes. This paper is likely to be highly cited in subsequent hydropowermanagement work as well as in the RL community.