A Pseudo-Euclidean Iteration for Optimal Recovery in Noisy ICA
Independent Component Analysis (ICA) is a popular model for blind signal separation. The ICA model assumes that a number of independent source signals are linearly mixed to form the observed signals. We propose a new algorithm, PEGI (for pseudo-Euclidean Gradient Iteration), for provable model recovery for ICA with Gaussian noise. The main technical innovation of the algorithm is to use a fixed point iteration in a pseudo-Euclidean (indefinite "inner product") space. The use of this indefinite "inner product" resolves technical issues common to several existing algorithms for noisy ICA. This leads to an algorithm which is conceptually simple, efficient and accurate in testing. Our second contribution is combining PEGI with the analysis of objectives for optimal recovery in the noisy ICA model. It has been observed that the direct approach of demixing with the inverse of the mixing matrix is suboptimal for signal recovery in terms of the natural Signal to Interference plus Noise Ratio (SINR) criterion. There have been several partial solutions proposed in the ICA literature. It turns out that any solution to the mixing matrix reconstruction problem can be used to construct an SINR-optimal ICA demixing, despite the fact that SINR itself cannot be computed from data. That allows us to obtain a practical and provably SINR-optimal recovery method for ICA with arbitrary Gaussian noise.
This paper addresses the problem of blind signal extraction (recovery) in the presence of Gaussian noise.To solve the noisy ICA problem the authors employ a fixed point iteration to achieve a maximization of directional kurtosis.The fixed-point iteration is carried out in a pseudo-Euclidean space.Based on the proposed "pseudo-Euclidean gradient iteration (PEGI)" two algorithms are presented: one for estimating a single column of the mixing matrix and a second one using an adapted deflation method to find the full mixing matrix while making sure not to find the same component again.clarity:The paper is clearly written and a pleasure to read. However some key references of related work are missing (see below).originality:The approach is similar to the well-known fastICAmethod and the GI-ICA [21] (Voss et al.,NIPS 2013), however it differs from those methods in an important aspect: It does neither rely on the pre-whitening (orthogonalization) step nor on "quasi-whithening" (quasi-orthogonalization).The main contributions(1)gradient-based fixed point iterations for kurtosis maximisation are done in a pseudo-Euclidean space(2) analysis and insight how to choose this pseudo-Euclidean space optimally for the noisy ICA caseare technically sound and sufficiently novel. Significance:Performing ICA in the noise-corrupted setting is an important problem of potential interest to the NIPS crowd.Though the robustness to Gaussian noise is inherent to all higher-order cumulant-based approaches and is well know, the proposed method to integrate this into a fixed-point algorithm is admittedly elegant and thus the contribution can be considered significant. A further plus is the straightforward extension to the complex case (which is shown in supplemental material) The experimental section is a bit weak, only synthetic data is considered and many other robust ICA methods are not included in comparison.At least experimental results demonstrating improvements over GI-ICA indicate that the cumbersome quasi-orthogonalization as in [21] is no longer needed.On the other hand there exist many methods for the noisy ICA problem. In particular, the algorithm in [R2] "RobustICA" would be the real competitor, because as shown in [R3] the fixed-point algorithm can be seen as a special case of gradient descent with an optimally selected stepsize as proposed in [R2] but such a comparison is not included. Also the relative Newton method of Zibulevsky [R5] for minimization of a quasi-ML cost function would have provided a better baseline.A further issue is that the estimation of 4-th order cumulants / kurtosis in general needs a LOT of data (1e+5--1e+6 samples are often needed, as can be seen from figure 2)Furthermore the practical usefulness for real-world data remains to be demonstrated.further comments:-It would be recommended to compare fastICA with pow3 not tanh nonlinearity in Fig1-Why is the iteration index k omitted in algorithm 2 ? additional references:[R1] Sergio Cruces, A. Cichocki, S. Amari, "The Minimum Entropy and Cumulant Based Contrast Functions for Blind Source Extraction", Lecture Notes in Computer Science LNCS-2085, Springer-Verlag, pp. 786-793, 2001.[R2] Vicente Zarzoso and Pierre Comon, "Robust Independent Component Analysis by Iterative Maximization of the Kurtosis Contrast With Algebraic Optimal Step Size" IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 21, NO. 2, FEBRUARY 2010,10.1109/TNN.2009.2035920 [R3] H. Li and T. Adali. A class of complex ICA algorithms based on the kurtosis cost function.IEEE Transactions on Neural Networks, 19(3):408-420, 2008. [R4] Javidi S, Mandic DP, Took CC, and Cichocki A: "Kurtosis-based blind source extraction of complex non-circular signals with application in EEG artifact removal in real-time.", Front Neurosci, 5, 105 (2011)[R5] M. Zibulevsky,"Relative Newton and Smoothing Multiplier Optimization Methods for Blind Source Separation", chapter in the book: S. Makino, T.W. Lee and H. Sawada eds., Blind Speech Separation, Springer Series: Signals and Communication Technology XV, 2007[R6] P. A. Regalia and E. Kofidis, "Monotonic convergence of fixed-point algorithms for ICA," IEEE Trans. Neural Netw., vol. 14, no. 4, pp. 943-949, Jul. 2003.summaryThe authors present an elegant modification of the fixed-point iteration for robust ICA in a kurtosis-maximization framework.The provided insights would be of interest to the ICA community even though the practical usefulness of the method for real-world data remains to be demonstrated.