Asynchronous Parallel Coordinate Minimization for MAP Inference
Finding the maximum a-posteriori (MAP) assignment is a central task for structured prediction. Since modern applications give rise to very large structured problem instances, there is increasing need for efficient solvers. In this work we propose to improve the efficiency of coordinate-minimization-based dual-decomposition solvers by running their updates asynchronously in parallel. In this case messagepassing inference is performed by multiple processing units simultaneously without coordination, all reading and writing to shared memory. We analyze the convergence properties of the resulting algorithms and identify settings where speedup gains can be expected. Our numerical evaluations show that this approach indeed achieves significant speedups in common computer vision tasks.
SUMMARY:
========
The authors propose a parallel MAP inference algorithm based on a dual decomposition of the MAP objective.  Updates are performed without synchronization and locally enforce consistency between overlapping regions.  A convergence analysis establishes a linear rate of convergence and experiments are performed on synthetic MRFs and image disparity estimation.
PROS:
=====
The paper is well-rounded and provides algorithmic development, theoretical analysis in terms of convergence rate results, and experimental validation.  Clarity is good overall and the work is interesting, novel, and well-motivated.
CONS:
=====
Experimental validation in the current paper is inadequate in several respects.  First, since the approach is posed as a general MAP inference it would be ideal to see validation in more than one synthetic and one real world model (modulo variations in network topology).  Second, the size of problem instances is much too small to validate a "large-scale" inference algorithm.  Indeed, as the authors note (L:20-21) a motivation of the proposed approach is that "modern applications give rise to very large instances".  To be clear, the "size" of instances in this setting is the number of regions in Eq. (1).  The largest example is on image disparity, which contains less than 27K regions.  By contrast, the main comparison, HOGWILD, was originally validated on three standard "real world" instances with a number of objective terms reaching into the millions.  Note that even single-threaded inference converges within a few hundred seconds in most cases.  Finally, the authors do not state which disparity estimation dataset and model are used.  This reviewer assumes the standard Middlebury dataset it used, but the reported number of regions is much smaller than the typical image size in that dataset, please clarify.
Results of the convergence analysis are not sufficient to justify the lack of experimental validation.  In particular, Theorem 1 provides a linear rate of convergence, but this does not establish global linear rate of convergence as the bound is only valid away from a stationary point (e.g. when the gradient norm is above threshold).  It is also unclear to what extent assumptions of the convergence rate can be evaluated since they must be such that the delay rate, which is unknown and not controlled, is bounded above as a function of c and M.  As a secondary issue, this reviewer is unclear about the sequential results.  Specifically, it is not clear why the upper bound in Prop. 2 is necessary when an exact calculation in Prop. 1 has been established, and as the authors point out this bound can be quite loose.  Also, it is not clear what the authors mean when they state Prop. 2 accounts for the sparsity of updates.  Please clarify.