Deep Joint Task Learning for Generic Object Extraction
This paper investigates how to extract objects-of-interest without relying on handcraft features and sliding windows approaches, that aims to jointly solve two subtasks: (i) rapidly localizing salient objects from images, and (ii) accurately segmenting the objects based on the localizations. We present a general joint task learning framework, in which each task (either object localization or object segmentation) is tackled via a multi-layer convolutional neural network, and the two networks work collaboratively to boost performance. In particular, we propose to incorporate latent variables bridging the two networks in a joint optimization manner. The first network directly predicts the positions and scales of salient objects from raw images, and the latent variables adjust the object localizations to feed the second network that produces pixelwise object masks. An EM-type method is presented for the optimization, iterating with two steps: (i) by using the two networks, it estimates the latent variables by employing an MCMC-based sampling method; (ii) it optimizes the parameters of the two networks unitedly via back propagation, with the fixed latent variables. Extensive experiments suggest that our framework significantly outperforms other state-of-the-art approaches in both accuracy and efficiency (e.g. 1000 times faster than competing approaches).
Summary:This paper proposes a model that jointly trains two networks. One network triesto do localization. The second network does segmentation.  The first networkoutputs the coordinates of a bounding box. The image is cropped using thesecoordinates and resized to a fixed square size. The resulting image is given as inputto the second network. The second network predicts a pixel-wise binary mask tosegment out the foreground. The model is supervised i.e., pixel-wise segmentationlabels are given.  The authors argue that if the two tasks are decoupled and atight bounding box around the object is used as the target for the firstnetwork, then the predicted bounding boxes may be too tight and not provideenough context to do segmentation. They may also cut across some region of theforeground.To prevent this, the authors introduce latent variables which are perturbationsin the coordinates of the bounding box. The localization net now has to predictthe perturbed bounding box. However the optimal perturbation is not known.Therefore the model optimizes for the perturbation using a Metropolis-Hastings sampling strategy that uses super-pixels to figure out the goodness of any perturbation.Strengths- Shows improvements over decoupling the tasks and using no perturbation.- The model is very efficient.- The performance is compared to many other models. These models include ones based on hand-crafted features alone as well as ones that involve some form of learning. The model is convincingly better than these other models.Weaknesses- A simple baseline would be to just add a 5 or 10 pixel boundary around the predictions of the localization net. The authors do not compare to this baseline.- A slightly stronger baseline would be to modify the predicted bounding boxes using super-pixels as a post-processing step (maybe using the same idea that is used in the data-dependent term of the MH proposal distribution). This baseline is not used either.- In the absence of strong baselines, it is not clear what the joint optimization buys us.- The proposal distribution's data-dependent term uses super-pixel cues in a very ad-hoc way. The term is designed to avoid cutting through super-pixel regions. However, unless the super-pixel regions are aligned with the axes of the bounding box rectangle, it seems that it would be hard to avoid cutting through super-pixels. Moreover, there may be a lot of background region going across the edges of the bounding box. It is perfectly reasonable for the bounding box to cut through super-pixels in such regions.Quality:The paper does not compare to simpler baseline models (as mentioned in the weaknesses section) which makes it hard to assess it contribution.Clarity:There seem to be many typos and grammatical errors, some of which are mentioned here.Minor typos -- Eq(1) The equality should probably be a proportionality.- Eq(2) There seems to a square missing over the L2 norm.- '... may lead to unsatisfied pixel-wise segmentation ..' unsatisfied -> unsatisfactory ?Originality:The idea of using Metropolis-Hastings to combine two tasks seems interesting. However, this particular application is not very compelling.Significance:The main idea is interesting but more experiments with stronger baselines are needed to assess the potential impact of this model.