Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation
Many methods have been proposed to solve the problems of recovering intrinsic scene properties such as shape, reflectance and illumination from a single image, and object class segmentation separately. While these two problems are mutually informative, in the past not many papers have addressed this topic. In this work we explore such joint estimation of intrinsic scene properties recovered from an image, together with the estimation of the objects and attributes present in the scene. In this way, our unified framework is able to capture the correlations between intrinsic properties (reflectance, shape, illumination), objects (table, tv-monitor), and materials (wooden, plastic) in a given scene. For example, our model is able to enforce the condition that if a set of pixels take same object label, e.g. table, most likely those pixels would receive similar reflectance values. We cast the problem in an energy minimization framework and demonstrate the qualitative and quantitative improvement in the overall accuracy on the NYU and Pascal datasets.
This paper is very exciting. The authors present a 
      technique for jointly inferring the intrinsic properties of a scene 
      (shapes, colors, illuminations) and a semantic-segmentation of a scene 
      (objects labels, attributes). The technique basically works by gluing 
      together Barron & Malik's SIRFS work with the DenseCRF work, where the 
      "glue" holding the two models together is a dual-decomposition technique 
      built around a segmentation technique. Doing these two tasks jointly seems 
      to help both tasks --- not by a great deal, but probably enough. 
      The goal of this paper is great. Combining semantics and 
      intrinsics is definitely something the computer vision community should 
      focus on more, and I can see the machine learning community getting 
      excited about the ideas here. The algorithm, though necessarily very 
      complicated as it's a superset of complicated models, is presented well. 
      The idea is novel, as is the overall model, though the machinery is not 
      very novel itself. I do have some concerns regarding clarity and 
      evaluation. First, it's not clear to me how to contrast this model to 
      Barron & Malik's stuff. Though the authors cite [4] (the scene-based 
      version of the model) they don't seem to have used it. Instead they've 
      made a new "scene" generalization out out what is presumably some past 
      version of SIRFS. Can the authors clarify exactly what they did? It seems 
      like the authors used the technique from [2] on different segments of the 
      image, while sharing illumination across the entire image. This seems to 
      be a technique for generalizing SIRFS to scenes, but it is not [4]. How 
      does this technique work by itself for the task in [4], of just doing 
      intrinsic reconstruction? Any why is only one illumination used? In [4] 
      they get a lot out of modeling a mixture of lights, and many of the 
      mistakes made by this model seem to be due to this incorrect assumption of 
      a single global model of illumination. This needs to be explained well, 
      because the author's "scene" version of [2] is used as a baseline 
      technique by this paper, when it seems to be a novel algorithm of its own. 
      Why not actually benchmark against [4]? The output here looks 
      significantly worse than that of [4]. One completely unmentioned 
      and very worrisome aspect of this paper is that in the NYU experiments 
      seem to use a Kinect depth map as input. I assume this is true just 
      because the output depth maps look too good to have been produced blind, 
      by either this paper or by SIRFS. How is the Kinect being used? And what 
      is being done in the PASCAL experiments in which there is not input depth? 
      What sort of shape reconstructions are produced on those images? Some 
      discussion of this is very necessary. I'm confused by the 
      comparison to [10]. The improvement of this paper with respect to [10] 
      seems somewhat small. I appreciate the author's honesty in submitting a 
      comparison to what is presumably another work of theirs, but why was this 
      done? Are components of this model shared between these two works? If so, 
      what are they? And does [10] use extra "verbal" information, as the 
      attached paper suggests? Thankfully, the comparison against [12] seems 
      correct, and this paper does improve the results by a medium-sized amount, 
      which is enough to convince me that this technique works. My 
      greatest concern is that the improvement of this paper over [12] may be 
      entirely due to the use of the Kinect depth input. It's possible that 
      SIRFS is serving to do nothing except indirectly inform the 
      attribute/object CRF components of the shape of the scene, which may be 
      the only driving force behind the improved numbers. To verify that this is 
      not the case, a simple baseline in which the DenseCRF also has access to 
      the Kinect depth map (in some simple featurized form, of which several now 
      exist) should also be presented. Overall, I think this paper 
      deserves credit for attempting an extremely ambitious task. The 
      qualitative improvement (looking at the intrinsic output) is not very 
      large, and the quantitative semantic improvement looks okay, but not very 
      significant. I would really like my concerns regarding evaluation and the 
      Kinect input to be addressed, as I feel like this will completely 
      determine how I feel about the true contribution of this 
      paper.