Alternating Minimization for Regression Problems with Vector-valued Outputs
¬¸»² ©» ©·´´¸¿ª» µ© © µ î ã°²÷ò Ø±©»ª»®ô «¾¬®¿½¬·²¹ ¬¸» ¬©± »¯«¿¬·±² ¹·ª»ae § ï § î ã øÈ ï È î ÷ â © å ï · ²ò Ì¸¿¬ ·ô ¿ ±±² ¿ ©»¸¿ª» ² ¼ ¿³°´»ô ©» ©·´® »½±ª»® © »¨¿½¬´ § ¾ § ±´ª·²¹ ¬¸» ¿¾±ª» §¬»³ ±º´·²»¿® »¯«¿¬·±²ÿ Ñ«® ¬± § »¨¿³°´» ³±¬·ª¿¬» ¬¸» º«²¼¿³»²¬¿´¯«»¬·±² ¬¸¿¬ ¬¸·°¿°»® ¿²©»®ae¸±© ³«½¸½¿² ©» ·³°®±ª» ÑÔÍ ¾ § »¨°´±·¬·²¹ ²±·» ½±®®»´¿¬·±²á Ô»¬ « ³¿µ» ¬¸» »¨¿³°´» ³±®» ®»¿´·¬·½ ¾ § ¿«³·²¹ ¬¸» ³±¼»´ae § ã È © õ å ï · ²ô ©¸»®» § î Î · ¿ ª»½¬±® ±º ³ ®»°±²»ô »¿½¸»´»³»²¬ ±º È î Î · ¿³°´»¼ ·ò·ò¼ò º®±³ ¬¸» ¬¿²¼¿®¼ Ù¿«·¿² ¿²¼ ²±·» ª»½¬±® · ¼®¿©² º®±³ Ò øðå ÷ò ß ½±®±´´¿® § ±º ¬¸» ³¿·² ®»«´¬ ·² Í»½¬·±² î ¸±© ¬¸¿¬ ÓÔÛ ø»» øí÷÷ ·³°®±ª» «°±² ¬¸» ÑÔÍ°¿®¿³»¬»® »®®±® ¾±«²¼ ¾ § ¿ º¿½¬±® ±º Û®®±® ãÛ®®±® ï ÷ã³ î ò Ì¸· º¿½¬±® ½¿² »¿·´ § ¾» »»² ¬± ¾»´¿®¹»® ¬¸¿² ï ¾ § «·²¹ Ý¿«½¸ §óÍ½¸©¿®¦ ·²»¯«¿´·¬ §ae ï ÷ã³ î ã ø ÷ø ï ÷ã³ î ø ïã ÷ î ã³ î ã ïô ©¸»®» ¾» ¬¸» ¶ó¬¸´¿®¹»¬ »·¹»²ª¿´«» ±º ò Ì¸» ·²»¯«¿´·¬ § · ¬·¹¸¬ ©¸»² ã ½ã º±® ±³» ½±²¬¿²¬ ½ò Ì¸¿¬ ·ô ©¸»² ã ½× ©¸·½¸¸±´¼ ¬®«» ·ºº ¬¸» ²±·» ·² »¿½¸®»°±²» · ³«¬«¿´´ § ·²¼»°»²¼»²¬ ¿²¼¸¿ ¿³» ª¿®·¿²½»ò Ì¸» ³±®» ¼»°¿®¬ º®±³ ¾»·²¹ ½ ×ô ¬¸»´¿®¹»® ¬¸» ·³°®±ª»³»²¬ º¿½¬±®ò Ú±® »¨¿³°´»ô ½±²·¼»® ³ ã î ½¿» ¿¹¿·²ô ¾«¬ ®¿¬¸»® ¬¸¿² ï ã î ô ©»¸¿ª»¸·¹¸´ § ½±®®»´¿¬»¼ Å ï î Ã ©·¬¸½±ª¿®·¿²½» ³¿¬®·¨ã   
Í»» ß°°»²¼·¨Ý º±® ¼»¬¿·´»¼°®±±º ±º ¾±¬¸±º ¬¸» ®»«´¬ ¹·ª»² ¿¾±ª»ò
É» ²±© ½±³¾·²» ¬¸» ¿¾±ª» ¹·ª»² ±¾»®ª¿¬·±² ©·¬¸Ì¸»±®»³ î ¬± ±¾¬¿·² ±«® »®®±® ¾±«²¼ º±® ß´¬Ó·²°® ±½»¼«®»ò Ë·²¹ ¿ ´·¹¸¬´ § ¬®±²¹»® ª»®·±² ±º Ì¸»±®»³ ïëô ©» ½¿² ¿´± ±¾¬¿·² ¬¸» »®®±® ¾±«²¼ º±® ¬¸» ÑÔÍ øÑ®¼·²¿® § Ô»¿¬ Í¯«¿®»÷ ±´«¬·±² ¿ ©»´´¬¸» ÓÔÛ ±´«¬·±²ò Ý±®±´´¿® § ì øÎ»«´¬ º±® Ð±±´»¼ Ó±¼»´ô Ù¿«·¿² Ü¿¬¿ô ×²¼»°»²¼»²¬ Î±©÷ò Ô»¬ È ¾» ¿³°´»¼ ò¬ò
Ô±©»® Þ±«²¼ º±® ÑÔÍ ¿²¼ ÓÔÛae É» ²±© ¸±© ¬¸¿¬ ¬¸» »®®±® ¾±«²¼ º±® ¾±¬¸¬¸» ÑÔÍ ¿ ©»´´¿ ¬¸» ÓÔÛ ±´«¬·±² ¬¿¬»¼ ¿¾±ª» ¿®» ·² º¿½¬ ¬·¹¸¬ «°¬±´±¹óº¿½¬±®ò Ô»³³¿ ëò Ô»¬ ¬¸» ¿«³°¬·±² ±º Ý±®±´´¿® § ì¸±´¼ò Ì¸»²ô ©»¸¿ª» ø©ò°ò ï ïã² 
Ë·²¹ Ì¸»±®»³ î ©·¬¸Ì¸»±®»³ ïë ¿²¼ øíî÷ ø©·¬¸½»®¬¿·² ßô Þ÷ ©» ±¾¬¿·² ¬¸» º±´´±©·²¹ ½±®±´´¿® §ò Ý±®±´´¿® § ê øÎ»«´¬ º±® Ð±±´»¼ Ó±¼»´ô Ù¿«·¿² Ü¿¬¿ô Ü»°»²¼»²¬ Î±©÷ò Ô»¬ È ¿¾±ª»ò Ô»¬ ² Ýø³ õ ¼÷´±¹ø³ õ ¼÷ò Ì¸»² ¬¸» º±´´±©·²¹¸±´¼ ø©ò°ò ï Ì ã² ïð ÷ae 
Ø»®» ¿¹¿·²ô ¬¸» ÓÔÛ°®±¾´»³ · ²±²ó½±²ª»¨¿²¼¸»²½» ¬¿²¼¿®¼ ¿²¿´ §· ¼±» ²±¬°®±ª·¼» ¬®±²¹ ½±²ª»®¹»²½» ¹«¿®¿²¬»»ò Í¬·´´ô ¿´¬»®²¿¬·²¹ ³·²·³·¦¿¬·±² ø±º ²»¹¿¬·ª»´±¹ó´·µ»´·¸±±¼÷ º±® É å´»¿¼ ê ß´¹±®·¬¸³ î ß´¬Ó·²óÍËÎae ß´¬»®²¿¬·²¹ Ó·²·³·¦¿¬·±² º±® ÍËÎ Î»¯«·®»ae Ü ã ºøÈ ï å § ï ÷ ae ae ae øÈ î å § î ÷¹ô Ò«³¾»® ±º ·¬»®¿¬·±²ae Ì ïae Î¿²¼±³´ §°¿®¬·¬·±² Ü ã ºÜ ð å Ü ð å Ü ï å Ü ï å ae ae ae å Ü å Ü ¹ô ©¸»®» ¶Ü © ¶ ã ¶Ü ¶ ã ²ô è¬ îae ×²·¬·¿´·¦» É ð ã ð íae º±® ¬ ã ðå ae ae ae å Ì ï ¼±
SUMMARY:This paper studies the effect of noise correlation in some models of multi-output regression. It argues that a method that does not benefit from the correlation, such as Ordinary Least Squares (OLS), may perform much worse than a method that does, such as Maximum Likelihood Estimation (MLE). For certain linear models (Pooled model and Seemingly Unrelated Regression), which are studied in the paper, the MLE estimator requires the joint optimization of the covariance and regression weights. This is a non-convex problem. Alternative Minimization (AltMin) algorithm is an approach to solve the problem by iteratively optimizing the covariance and the weights. This is not guaranteed to find the global optimum.The contribution of the paper is a finite-sample study of AltMin and MLE and showing the effect of correlation. The results indicate that both MLE and AltMin can in fact benefit from the correlation, but OLS cannot. For certain examples, the difference can be significant. The paper provides lower bound for OLS and MLE (for the Pooled model) that up to a logarithmic factor in the number of samples is the same as the upper bound. This indicates that the upper bound of AltMin, which is the same as MLE's, is indeed tight.The paper has some empirical studies too. EVALUATION:I believe this is a reasonably good paper. It provides intuition, has theory, and performs simple experiments that validate the theory. I have a few comments:- The eps in the experiments is very small, indicating highly correlated outputs. Would the difference between MLE/AltMin and OLS be still significant when eps is much larger (maybe in the order of 0.1-0.5)? The theory suggests that the behaviour is O(eps), but what about the effect of other constants, which might be different between OLS and AltMin/MLE?- The paper assumes that the noise model is Gaussian. How crucial is this for the conclusions of the paper? It appears that the only step that requires this Gaussian assumption is Lemma 11, which is Corollary 5.35 of [28]. It seems that one can safely use results of Section 5.3.2 of [28] to extend the results to beyond Gaussian noise models. Is this correct?- Some typos in the proof of Theorem 15 (p13):+ L650: The summation over i is missing in the RHS. + L656: zeta_1 is on top of the inequality symbol, but is never referred to. + L663: It seems that a log n term is missing after the equality. - The proof of Lemma 3, L710: The inequality indicated by zeta_1 holds with a probability at least 1 - exp(-cn), but it is not stated and taken care of. ===== UPDATE: Thank you. You addressed my concerns.