OPTIMAL CONTROL VIA NEURAL NETWORKS: A CONVEX APPROACH
Control of complex systems involves both system identification and controller design. Deep neural networks have proven to be successful in many identification tasks, however, from model-based control perspective, these networks are difficult to work with because they are typically nonlinear and nonconvex. Therefore many systems are still identified and controlled based on simple linear models despite their poor representation capability. In this paper we bridge the gap between model accuracy and control tractability faced by neural networks, by explicitly constructing networks that are convex with respect to their inputs. We show that these input convex networks can be trained to obtain accurate models of complex physical systems. In particular, we design input convex recurrent neural networks to capture temporal behavior of dynamical systems. Then optimal controllers can be achieved via solving a convex model predictive control problem. Experiment results demonstrate the good potential of the proposed input convex neural network based approach in a variety of control applications. In particular we show that in the MuJoCo locomotion tasks, we could achieve over 10% higher performance using 5× less time compared with state-of-the-art model-based reinforcement learning method; and in the building HVAC control example, our method achieved up to 20% energy reduction compared with classic linear models.
This is a well-motived paper that considers bridging the gap
in discrete-time continuous-state/action optimal control
by approximating the system dynamics with a convex model class.
The convex model class has more representational power than
linear model classes while likely being more tractable and
stable than non-convex model classes.
They show empirical results in Mujoco continuous-control
environments and in an HVAC example.
I think this setup is a promising direction but I have
significant concerns with some of the details and claims
in this work:
1. Proposition 2 is wrong and the proposed input-convex recurrent
   neural network architecture not input-convex.
   To fix this, the D1 parameters should also be non-negative.
   To show why the proposition is wrong, consider the convexity of y2
   with respect to x1, using g to denote the activation function:
       z1 = g(U x1 + ...)
       y2 = g(D1 z1 + ...)
   Thus making
       y2 = g(D1 g(U x1 + ...) + ...)
   y2 is *not* necessarily convex with respect to x1 because D1 takes
   an unrestricted weighted sum of the convex functions g(U x1 + ...)
   With the ICRNN architecture as described in the paper not being
   input-convex, I do not know how to interpret the empirical findings
   in Section 4.2 that use this architecture.
2. I think a stronger and more formal argument should be used to show
   that Equation (5) is a convex optimization problem as claimed.
   It has arbitrary convex functions on the equality constraints that
   are composed with each other and then used in the objective.
   Even with parts of the objective being convex and non-decreasing
   as the text mentions, it's not clear that this is sufficient when
   combined with the composed functions in the constraints.
3. I have similar concerns with the convexity of Equation (6).
   Consider the convexity of x3 with respect to u1, where g is
   now an input-convex neural network (that is not recurrent):
       x3 = g(g(x1, u1), u2)
   
   This composes two convex functions that do *not* have non-decreasing
   properties and therefore introduces an equality constraint that
   is not necessarily even convex, almost certainly making the domain
   of this problem non-convex. I think a similar argument can be
   used to show why Equation (5) is not convex.
In addition to these significant concerns, I have a few other
minor comments.
1. Figure 1 hides too much information. It would be useful to know,
   for example, that the ICNN portion at the bottom right
   is solving a control optimization problem with an ICNN as
   part of the constraints.
2. The theoretical results in Section 3 seem slightly out-of-place within
   the broader context of this paper but are perhaps of standalone interest.
   Due to my concerns above I did not go into the details in this portion.
3. I think more information should be added to the last paragraph of
   Section 1 as it's claimed that the representational power of
   ICNNs and "a nice mathematical property" help improve the
   computational time of the method, but it's not clear why
   this is and this connection is not made anywhere else in the paper.
4. What method are you using to solve the control problems in
   Eq (5) and (6)?
5. The empirical setup and tasks seems identical to [Nagabandi et al.].
   Figure 3 directly compares to the K=100 case of their method.
   Why does Fig 6 of [Nagabandi et al.] have significantly higher rewards
   for their method, even in the K=5 case?
6. In Figure 5, f_NN seems surprisingly bad in the red region of the
   data on the left side. Is this because the model is not using
   many parameters? What are the sizes of the networks used?