Fast Training of Pose Detectors in the Fourier Domain
In many datasets, the samples are related by a known image transformation, such as rotation, or a repeatable non-rigid deformation. This applies to both datasets with the same objects under different viewpoints, and datasets augmented with virtual samples. Such datasets possess a high degree of redundancy, because geometrically-induced transformations should preserve intrinsic properties of the objects. Likewise, ensembles of classifiers used for pose estimation should also share many characteristics, since they are related by a geometric transformation. By assuming that this transformation is norm-preserving and cyclic, we propose a closed-form solution in the Fourier domain that can eliminate most redundancies. It can leverage off-the-shelf solvers with no modification (e.g. libsvm), and train several pose classifiers simultaneously at no extra cost. Our experiments show that training a sliding-window object detector and pose estimator can be sped up by orders of magnitude, for transformations as diverse as planar rotation, the walking motion of pedestrians, and out-of-plane rotations of cars.
This work presents a method to efficiently train object detectors in the presence of geometric transformations that can be represented as vector-matrix multiplications. This has recently been developed for the case of translation transformations ([1,8,14]) but has not been too obvious for other transformations, such as rotations, let alone non-rigid deformations.  The authors propose to adapt the Fourier-based method that originally allowed the development of efficient algorithms for the case of translations so that we can now deal with rotations and other `cyclic' signal transformations (e.g. the walking pattern of a pedestrian). The condition for this to hold is that the  transformation is norm-preserving, can be represented as a matrix multiplication, x_transformed = Q x,  has an inverse Q^{-1} = Q^T and for some s Q^s = I.  The starting point for the previous works was the fact that the 'data matrix' obtained by 'stacking' together all translated versions of a signal is a circulant  NxN matrix, where N is the length of the signal, and as such can be diagonalized using the discrete harmonic basis (or, Discrete Fourier Transform-DFT matrix), Eq. 3, and reference [5].The authors  note that in the case of general transforms (not translations) the 'data matrix' may not be circulant in general - but as long as the conditions above hold, the Gram matrix will be.  As such, one can exploit the DFT-based diagonalization to efficiently train a detector. Using these, the authors obtain closed-form solutions for ridge regression that only involve inversions of diagonal Gram matrixes  - and present also a nice extension for the simultaneous training of multiple pose detectors. Turning to applying this idea to visual data, we are presented with two different methods for constructing the Gram matrix, one involving an analytic construction of the transformation matrix Q, and another being more `data-driven', where there is in principle no underlying linear transformation, but rather a cyclic pattern, as in the walking cycle of pedestrians, or even out-of-plane rotations of cars. Result-wise the authors demonstrate remarkable acceleration in terms of training time, albeit with an occasional loss in performance, even when comparing to a ridge regression baseline.   This work presents a method to efficiently train object detectors in the presence of geometric transformations that can be represented as vector-matrix multiplications. This has recently been developed for the case of translation transformations ([1,8,14]) but has not been too obvious for other transformations, such as rotations, let alone non-rigid deformations.  The authors propose to adapt the Fourier-based method that originally allowed the development of efficient algorithms for the case of translations so that we can now deal with rotations and other `cyclic' signal transformations (e.g. the walking pattern of a pedestrian). The condition for this to hold is that the  transformation is norm-preserving, can be represented as a matrix multiplication, x_transformed = Q x,  has an inverse Q^{-1} = Q^T and for some s Q^s = I.  The starting point for the previous works was the fact that the 'data matrix' obtained by 'stacking' together all translated versions of a signal is a circulant  NxN matrix, where N is the length of the signal, and as such can be diagonalized using the discrete harmonic basis (or, Discrete Fourier Transform-DFT matrix), Eq. 3, and reference [5].The authors  note that in the case of general transforms (not translations) the 'data matrix' may not be circulant in general - but as long as the conditions above hold, the Gram matrix will be.  As such, one can exploit the DFT-based diagonalization to efficiently train a detector. Using these, the authors obtain closed-form solutions for ridge regression that only involve inversions of diagonal Gram matrixes  - and present also a nice extension for the simultaneous training of multiple pose detectors. Turning to applying this idea to visual data, we are presented with two different methods for constructing the Gram matrix, one involving an analytic construction of the transformation matrix Q, and another being more `data-driven', where there is in principle no underlying linear transformation, but rather a cyclic pattern, as in the walking cycle of pedestrians, or even out-of-plane rotations of cars. Result-wise the authors demonstrate remarkable acceleration in terms of training time, albeit with an occasional loss in performance, even when comparing to a ridge regression baseline.   There is in my understanding a substantial amount of originality in this paper, proposing to combine recent advances in fast learning of object detectors in the presence of translations, with more challenging transformations. Some 'out of the box' thinking was required here, as it is most common to deal with rotations through polar coordinates - which would make everything harder; it turns out that by using the Gram matrix we are back to the setting of [14], which is a not-too-evident result; so I was very interested when originally reading the paper.   Still, the paper leaves more to be desired to make the paper significant to the learning/vision community. The most obvious critique is that the detection results are not compared to a stronger baseline, e.g. an SVM-based detector, so we do not know to what extent the presented method will be practically useful. Along the same lines, the authors could build a pose-invariant detector out of their system (by using the max over poses, as they discuss) and compare it to the methods that are currently out there for detection in the presence of pose variation (e.g. [24] or [c] below).  It is also disappointing to see that there is no discussion about how this method could be extended to other problems, beyond ridge regression. In [14], and (a) below, a combination of similar ideas with the hinge loss has been pursued; it should at least be discussed, and ideally tried out.  This is, in my opinion, the most important development that would be needed to make this paper have high impact. As it stands, the work would be of use only to practitioners who want a 'quick and dirty' detector, which may be only as good as one trained with ridge regression. In particular it is not true that ridge regression (+ whitening) performs similar to SVMs for detection; it merely serves as an 'ok' proxy. This was made more pronounced  in the DPM training work of R. Girshick, J. Malik 'Training Deformable Part Models with Decorrelated Features' in ICCV 2013  - there was quite a big difference in performance with and without the final SVM training stage.  I found it also disappointing  that the presentation is a bit blurred,  leaves several gaps to be filled in by the reader's imagination. I provide below a (partial) list of  points where this was the case.  Eq. 3: The DFT matrix F is never defined; so even if the reader has a signal processing background, one still needs to think about the period of the DFT (is it s or m?) whether F^H is the inverse of forward DFT (I understand it is the inverse) and what the normalization factors are; a single line equation would have made all of this clear.    l. 157-158: 'the matrices involved in a learning problem become diagonal, which drastically reduces the needed computations': There are many learning problems, and many ways in which matrices may be involved. Unless the authors pin down their optimization problem, show how matrices are involved in it,  and clarify to what extent the computations become reduced, this is a vague statement.  l. 182: 'structure of the covariance matrix is unknown': we do know the structure of the covariance matrix (we have an equation for it in l. 174 - and it is diagonalizable); we simply cannot express it in terms of the DFT matrix. l. 182: 'the diagonalization trick ... offers a way out, by computing the Gram matrix, and solving the dual problem'->The Gram matrix can be computed anyway, and the dual problem can be solved irrespectively of the diagonalization trick. So maybe the authors should be a bit more precise, stating e.g. 'The diagonalization technique allows us to express the Gram matrix in a form which makes an analytical solution of the dual problem possible.'l. 190: 'a regularized risk': The authors have to provide the form of the regularized risk. It is very common indeed, but presentation-wise it cannot be that the single most important thing (the optimization objective) is never written down. Strictly speaking the paper does not make sense from here on, since there are no 'primal', 'dual' objectives provided anywhere, but the authors keep referring to them. Lack of space is not a real excuse, several more verbose parts of the text can easily become condensed. l. 191: why should we have 's' dual variables? Where do they show up? In general, it is bad practice to leave all technical details to other references [5]/[23], since this practically requires that the reader reads three papers rather than one. A paper needs to be as self-contained as possible, and we are talking about things that can be explained with a couple of equations. l. 198-205: this could be placed right after theorem 1. Here it  breaks the presentation's flow l. 209: the reason why Eq. 6 follows from applying the circulant decomposition to G is not entirely obvious: to a casual reader it may seem as if you were saying F(a/b) = F(a)/F(b), with F being the Fourier transform, which does not make too much sense. The result is correct indeed (it is a one line proof), but some hand-holding would help. Stating a few central properties of circulant matrices in Section 2.2 would be useful. Section 4.1: this is, in my understanding, the same story as [14] (they also pass through the gram matrix in their optimization, and finally end up solving many simpler problems). If this is not the case, the difference should be clarified, it it is, it should be stated.   Section 5.1: from the style of the presentation, it is not too clear what exactly the authors do; we read about 'possibilities' (l.299,l.322), but it would be better if the authors state in advance that we use 'such and such a method for this experiment because of this reason'  and then go on to describe what the method consists in. Section 5.3: I could not understand this section. Apparently the authors are describing a technical problem that they faced, but they describe it in a somehow esoteric way : 'we have no principled way of obtaining negative samples for different poses for the same transformation ... lacking a definitive answer a simple strategy is .. - we can further optimize this by realizing that..'I understand that it is about negative samples not coming with a pose annotation - but I would guess that if one knows the transformation (e.g. rotation) all one has to do is apply all transformations to each negative sample, and label all transformed samples as negatives - so I do not see what is special about negative samples.I cannot see why the samples are constant along the DFT dimension. The labels may be (all zeros), but the features will be changing if we transform the image.  l. 371: rephrasing is needed (break into sentences, describing the setup: Pascal criterion assigns ground truth object to hypothesis; pose error is measured with respect to that object, as #i/#poses where #i is discretized pose difference) Table 2: what do you mean by 'with calibration'? Was the proposed method not calibrated?  Appendix: to prove that the data matrix 'will not be circulant in general', we would also need a counterexample for the case of non-translations (I understand that there are plenty - but providing one is a formal requirement).  One work that seems to me relevant (in the sense that it is learning the transformation Q) is  [a] Transformation Equivariant Boltzmann Machines, Jyri J. Kivinen and Christopher K. I. Williams, ICANN 2011,   The authors should also cite a precursor to the more recent works [1,14]: [b] Maximum Margin Correlation Filter: A New Approach for Simultaneous Localization and Classification Andres Rodriguez, Vishnu Naresh Boddeti, B.V.K Vijaya Kumar and Abhijit Mahalanobis  IEEE Transactions on Image Processing 2013 Finally, this is one of the works working on detecting rotated cars, the authors could compare to it (or [28])  [c] Andrea Vedaldi, Matthew B. Blaschko, Andrew Zisserman: Learning equivariant structured output SVM regressors. ICCV 2011