Least Informative Dimensions
We present a novel non-parametric method for finding a subspace of stimulus features that contains all information about the response of a system. Our method generalizes similar approaches to this problem such as spike triggered average, spike triggered covariance, or maximally informative dimensions. Instead of maximizing the mutual information between features and responses directly, we use integral probability metrics in kernel Hilbert spaces to minimize the information between uninformative features and the combination of informative features and responses. Since estimators of these metrics access the data via kernels, are easy to compute, and exhibit good theoretical convergence properties, our method can easily be generalized to populations of neurons or spike patterns. By using a particular expansion of the mutual information, we can show that the informative features must contain all information if we can make the uninformative features independent of the rest.
This paper is devoted to the problem of finding 
      relevant stimulus features from the neural responses. The main idea is to 
      minimize the information between a certain set of features and the neural 
      responses combined with the remaining stimulus features in order to 
      separate the relevant from irrelevant features. The method builds on the 
      prior work by Fukumizu and colleagues, but unfortunately makes an 
      additional restriction that greatly diminishes the potential use of this 
      method. Specifically, the irrelevant and relevant dimensions need to be 
      statistically independent from each other within the stimulus ensemble. 
      While the authors argue that this is not a crucial assumption “because 
      many distributions are Gaussian for which the dependencies between U 
      [informative features] and V[the uninformative features] can be removed by 
      pre-whitening the stimulus data before training”, this assumption 
      essentially eliminates the novel contribution from this method. This is 
      because with correlated Gaussian stimuli one can use spike-triggered 
      covariance (STC) to find all features that are informative about the 
      neural response in one simple computational step that avoids the 
      non-convexity and complexity of the proposed optimization. Furthermore, 
      the STC method has recent been shown to apply to elliptically symmetric 
      stimuli (Samengo & Gollisch J Computational Neuroscience 2013). 
      Finally, the proposed method has to iteratively increase the number of 
      relevant features, whereas this is not needed in the STC method. 
      In terms of practical implementations, the method was only 
      demonstrated in this paper on white noise signals. Even in this simple 
      case, the performance exhibited substantial biases. For example, in Figure 
      1, right panel, a substantial component along the irrelevant subspace 
      remained after optimization. Note that it is not correct to claim that 
      (lines 286-888, and also lines 315-316) that “LID correctly identifies the 
      subspace (blue dashed) in which the two true filters (solid black) reside 
      since projections of the filters on the subspace closely resemble the 
      original filters. “ The point of this method is to *eliminate* irrelevant 
      dimensions, and this was not achieved. The superior convergence 
      properties of the proposed techniques relatively to spike triggered 
      average/covariance were not demonstrated. In fact, the method did not even 
      remove biases in the STA, which would be possible using the linear 
      approximation (Figure 3). Note that the Figure 3 description was 
      contradictory: stimuli were described as white noise, yet the footnote 
      stated that the data were not whitened. The use of information 
      minimization for recovering multiple features and correlated stimuli was 
      demonstrated in Fitzgerald et al. PLoS Computational Biology 2011, 
      “Second-order dimensionality reduction using minimum and maximum mutual 
      information models.” Line 234-235: it is not correct that MID 
      (also STC) “requires data that contains several spike responses to the 
      same stimulus” – one can optimize the relative entropy based on a sequence 
      of stimuli presented only once in either MID/STC methods. Lines 
      236-237: MID/STC can be used with spike patterns by defining events across 
      time bins or different neurons. The abstract was written in a 
      somewhat misleading manner, because it is not the information that is 
      being minimized, but a related a quantity in combination with crucial 
      assumptions that are not spelled out in the abstract. For example, that 
      kernels need to factorize. The last statement in the abstract is also not 
      clear “… if we can make the uninformative features independent of the 
      rest.” Here, it is not clear that this implies that inputs are essentially 
      required to be uncorrelated if any of the input features can be 
      informative for one of the neurons in a dataset.