Neural Interaction Transparency (NIT): Disentangling Learned Interactions for Improved Interpretability
Neural networks are known to model statistical interactions, but they entangle the interactions at intermediate hidden layers for shared representation learning. We propose a framework, Neural Interaction Transparency (NIT), that disentangles the shared learning across different interactions to obtain their intrinsic lower-order and interpretable structure. This is done through a novel regularizer that directly penalizes interaction order. We show that disentangling interactions reduces a feedforward neural network to a generalized additive model with interactions, which can lead to transparent models that perform comparably to the state-of-theart models. NIT is also flexible and efficient; it can learn generalized additive models with maximum K-order interactions by training only O(1) models.
The paper proposes a variation of feedforward neural networks that does not learn (potentially unwanted) non-additive interactions between variables above a certain order.  The variation includes a block-based architecture with a sparsifying layer and a regularizer that caps the interaction order within blocks.  If the max interaction order is <= 2, then all interactions can be easily visualized as independent plots.  The authors draw an analogy between this model and GAMs with interactions.  Pros:  - The paper is relevant to NIPS and reasonably well written.  - Combining blocking and soft-capping the interaction order is an interesting idea and a natural extension to GAMs.  - Some results are nice: the proposed model can reach the same performance but better visualizability than the competitors.  Cons:  - The definition of interactions in this paper is not equivalent to the one given    in [28], which affects both the main message and the results.  It is also    confusing.  - Evaluation is lacking, see below.   Major remarks:   - The paper focuses on non-additive interactions.  However, definition 1 captures additive interactions as well.     Consider for instance the function f(x) = x1 + x2 + x3 + x4, which has no non-additive interactions at all.  Also let all variables be non-negative.  One way to write f(x) as a feedforward is:        f(x) = 1*h1(x) + 1*h2(x) + 0       h1(x) = max{1*x1 + 1*x2 + 0, 0}       h2(x) = max{1*x3 + 1*x4 + 0, 0}     Although this network has no non-additive interactions at all, definition 1 would classify all variables as entangled.     In other words, the presence of a path of positive weights does not imply that there is a non-additive interaction.  That is, the definition is incorrect.  Indeed, the authors do not show that it is equivalent to the actual definition of non-additive interactions given in [28].     Now, both the regularizer and the results depend on this definition.  This implies that:     - the regularizer controls a *surrogate* (actually, an upper bound) of the # of non-additive interactions.     - the # of interactions reported in the results may overestimate the real # of non-additive interactions.     To summarize, unless I'm missing something, the method does not do what the authors claim it does, and the same goes for the experiments.  The appropriate claims should be toned down accordingly.    - In order to see if DI really works, two things should be checked: (i) the first sparsifying layer works, (ii) the cap on maximum interaction order works.     Point (ii) has been validated empirically with real-world datasets.     The same is not true for point (i). The synthetic dataset is extremely simple (it is sampled from an order 2 homogeneous polynomial).     This is not enough to show that the sparsifying layer, especially when coupled with the cap on the interaction order, works as intended.  It should be easy to show that it does by testing it on *several* of order K polynomials (with varying K), and checking that the first layer learns the intended independencies.     The experiments with realistic datasets do not help in this case, because the gold standard is unknown.   - I also have a few other (neither major nor minor) questions about the experiments:    - The output layer of the proposed method is linear.  Is this also the case for binary classification tasks?    - I am a bit puzzled that there is no comparison to kernel methods (where the interaction order can be capped very easily, e.g. with polynomial kernels). Why were they left out of the comparison?    - It seems that training these models may be possibly complicated, potentially hindering reproducibility.  Will the code/data be made available?    - What about run-time comparisons?   Minor remarks:   - Figure 1 is very confusing.  I suggest to use colors to identify different models/regularizers and linestyles for the different layers.  The legend should also be sorted accordingly.   - l 110: a function the in the form   - l 152: to learn such form of model -> to learn such a model   - l 194: "since neural networks are known to learn smooth functions".  This may be the case in general, but it is definitely *not* the case for the ReLU networks used in this paper.  It is well known that ReLU networks with linear outputs learn piecewise linear functions (see [Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution, Chu et al., 2018] for a nice visualization), which are definitely not smooth.  The learned function may rensamble a smooth function, but this is only the case if the target function itself is smooth.  So the claim that the proposed method encourages smooth plots should be either removed or qualified appropriately.   - l 217: set to a small value [and] then increased   - l 216: "if K was not given ... model".  This is very confusing.  Please just    say that you report the performance of models while varying K.  (Otherwise it    sounds like a single value of K was chosen unfairly.)  ==== After reading the rebuttal, I decided to increase my score.  The issues I found can be either solved by rewording the paper (namely clarifying that the method uses a surrogate upper-bound of the # of non-additive independencies), or are not too major (they concern "only" the first layer).