Relevant sparse codes with variational information bottleneck
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y , while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y .
This paper combines the information bottleneck method (IB) approach with sparse coding. It proposes two instantiations of IB: first one is using sparse priors on the internal representation, and the second one is a kernel extension of the first model. A variational algorithm for learning the model is proposed, and the model is evaluated on simulated and real data (handwritten digits).
I find the paper novel and interesting. To my knowledge the algorithm is original and it adds to the existing tollbox of IB based approaches. The proposed method seems to outperform Gaussian IB on denoising and occlusion/inpaiting tasks on simulated and real data. It also provides new analysis tools for sparse representations in the form of IB information curves. Overall I think this work has many promising applications in machine learning and neuroscience and would be of interest to the NIPS audience.  Below I list a few questions / suggestions which could help authors to further improve the quality/readability of the paper:  1) As I understand, encoding/decoding dictionaries $W, U$ are trained for a fixed constraint $\gamma$. What is the influence of $\gamma$ on trained dictionaries? Do different receptive fields emerge under different constraints?  2) The authors validate the model using simulated data and handwritten digits. While these are necessary tests which demonstrate the correctness of the method, used datasets are somewhat constrained. It would be interesting to see the algorithm validated on a more diverse set of stimuli e.g. natural image patches.  Alternatively, I think that the authors should provide a brief justification for their choice of data (handwritten digits).   3) Perhaps it would be good to include a brief justification for the choice of the sparse prior. Why did the authors choose Student-t distribution? Is it analytical convenience, or are there some other reasons? Why not a generalized Gaussian for instance? Different priors have different entropies, which as I understand should affect the performance of the model.  4) I like the discussion which relates the proposed model to previous infomax/sparse coding approaches. Can we understand them as special cases of the sparse IB model?   Minor comments:  While overall the paper is very clearly written I have a feeling that in a few cases an additional line of explanation would substantially improve reception. For instance in equation (5) - it is not immediately obvious what is the relationship between the upper bound on I(X;R) and the log-likelihood of R.  Line 38 - the objective is to minimize the Lagrangian L (in the form the authors present it), not maximize it  Line 44 - I do not understand that sentence - what does "and" refer to?, Also - the inequality within the sentence could be given some more context for clarity.  Figure 2 D - as I understand "stimuli" on that panel means Y, not X. Perhaps you could clarify that.