  In this paper we investigate the use of reinforcement-learning based
prediction approaches for a real drinking-water treatment plant. Developing
such a prediction system is a critical step on the path to optimizing and
automating water treatment. Before that, there are many questions to answer
about the predictability of the data, suitable neural network architectures,
how to overcome partial observability and more. We first describe this dataset,
and highlight challenges with seasonality, nonstationarity, partial
observability, and heterogeneity across sensors and operation modes of the
plant. We then describe General Value Function (GVF) predictions -- discounted
cumulative sums of observations -- and highlight why they might be preferable
to classical n-step predictions common in time series prediction. We discuss
how to use offline data to appropriately pre-train our temporal difference
learning (TD) agents that learn these GVF predictions, including how to select
hyperparameters for online fine-tuning in deployment. We find that the
TD-prediction agent obtains an overall lower normalized mean-squared error than
the n-step prediction agent. Finally, we show the importance of learning in
deployment, by comparing a TD agent trained purely offline with no online
updating to a TD agent that learns online. This final result is one of the
first to motivate the importance of adapting predictions in real-time, for
non-stationary high-volume systems in the real world.

Machine Learning;Artificial Intelligence