We propose a method to impose homogeneous linear inequality constraints of the form Ax ≤ 0 on neural network activations. The proposed method allows a datadriven training approach to be combined with modeling prior knowledge about the task. One way to achieve this task is by means of a projection step at test time after unconstrained training. However, this is an expensive operation. By directly incorporating the constraints into the architecture, we can significantly speed-up inference at test time; for instance, our experiments show a speed-up of up to two orders of magnitude over a projection method. Our algorithm computes a suitable parameterization of the feasible set at initialization and uses standard variants of stochastic gradient descent to find solutions to the constrained network. Thus, the modeling constraints are always satisfied during training. Crucially, our approach avoids to solve an optimization problem at each training step or to manually trade-off data and constraint fidelity with additional hyperparameters. We consider constrained generative modeling as an important application domain and experimentally demonstrate the proposed method by constraining a variational autoencoder.
This paper proposes a method to impose linear inequality constraints on neural network activations. The method is implemented at initialization (by converting the H-representation to the V-representation) and during training (by modifying the network architecture). Experiments on two setups (projection and VAE+projection on a checkerboard pattern on MNIST) demonstrate a 2-orders of magnitude speed-up with respect to test-time projection (computed with OSQP).
The contributions claimed are:
* Novel technique to impose inequality constraints on neural network activations.
* Significant speed-up w.r.t. other techniques (at test time).
Overall, the approach is well motivated, and well-placed in the literature. However, the experimental analysis does not support all the claims made by the authors as it focuses on a single dataset (i.e., MNIST) and single constraint (i.e., checkerboard pattern). Additionally, while the authors argue that there is no manual trade-off between constraint satisfaction and data representation, the experiment in Fig. 3 appears to show that there is some manual trade-off (using box delay). As such, I am currently inclined to give a "weak reject" score.
The method is clear and the idea of using softmax to get a convex combination of the vertices of the V-representation to guarantee constraint satisfaction is reasonable.
1) The softmax used to satisfy the constraints is preceded by a batch normalization layer. I would expect batch normalization to interfere with the ability to saturate the softmax activation and thus prevent the network from reaching optimality. Could the authors provide experiments that justify the use of the batch normalization layer?
2) An important factor that is unexplored in this manuscript is the softmax temperature. A good scheduling of that temperature could help the optimization. Have authors tried different temperature values?
3) The use of softmax and the integration of the constraints as a network layer seem to create some difficulty during training (even with the rather simple checkerboard pattern used in the experiment). The loss appears to reach some plateau (9% from optimal) and, thus, there appears to be some trade-off between reconstruction and projection. The authors should provide more experiments to explain that trade-off. That trade-off is also visible on Fig. 5 (the zero has a significantly different shape).
4) The method is solely compared to test time projection. Could the authors implement other techniques (if feasible)? e.g., OptNet. Overall, it would helpful to add more setups and different types of constraints (other than a last-layer projection; e.g., monotonicity).