Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs
The idea of computer vision as the Bayesian inverse problem to computer graphics has a long history and an appealing elegance, but it has proved difficult to directly implement. Instead, most vision tasks are approached via complex bottom-up processing pipelines. Here we show that it is possible to write short, simple probabilistic graphics programs that define flexible generative models and to automatically invert them to interpret real-world images. Generative probabilistic graphics programs (GPGP) consist of a stochastic scene generator, a renderer based on graphics software, a stochastic likelihood model linking the renderer's output and the data, and latent variables that adjust the fidelity of the renderer and the tolerance of the likelihood. Representations and algorithms from computer graphics are used as the deterministic backbone for highly approximate and stochastic generative models. This formulation combines probabilistic programming, computer graphics, and approximate Bayesian computation, and depends only on generalpurpose, automatic inference techniques. We describe two applications: reading sequences of degraded and adversarially obscured characters, and inferring 3D road models from vehicle-mounted camera images. Each of the probabilistic graphics programs we present relies on under 20 lines of probabilistic code, and yields accurate, approximately Bayesian inferences about real-world images.
This paper proposes a general method for solving 
      image-based computer vision tasks using a generative probabilistic model 
      that uses a graphics program to generate images. The method takes the 
      standard Bayesian approach to frame the inference of the target variables, 
      and uses Metropolis-Hastings to perform the inference. This framework is 
      implemented for a CAPTCHA and a road-finding application, with favorable 
      results reported for each one. The primary contribution of this 
      paper is a proposal for using graphics programs as a key element of a 
      generative model for image-based tasks. While their claim that there are 
      no previous real-world image interpretation frameworks that combine 
      computer graphics among the other elements they list (last paragraph of 
      Section 1) seems accurate, their proposed system does not seem to qualify 
      as such a framework unless it's under a restricted interpretation. 
      Specifically, the CAPTCHA problem is "real-world" in one sense, but 
      CAPTCHAs are generated by a synthetic process and are therefore 
      qualitatively a very different type of problem than such real-world 
      problems as object recognition or detection in natural images. This 
      distinction is important, because one of the main reasons that 
      discriminative approaches have gained favor is due to their ability to 
      deal with complex, noisy features found in natural images. The authors 
      would presumably agree that their case is severely weakened if their 
      framework is limited to synthetic images such as CAPTCHAs. As for 
      the road-finding problem, it certainly qualifies as a real-world problem 
      and deals with natural images, but it is a strain to consider what they 
      use to be a graphics program. The elements that the model generates are 
      simply the properties (height, width, and lane size) of the lines that 
      form the lanes of the road, and the "renderer" is simply an assignment of 
      pixels to the regions delineated by these properties rather than an actual 
      realistic rendering of the scene. This is perfectly reasonable, but it is 
      no more of a computer graphics-based approach than the many previous 
      approaches that align geometric elements to a scene (e.g., [3], [8], Hedau 
      et al. ICCV '09, Lee et al. NIPS '10, Gupta et al. ECCV '10, and many 
      similar projects by some of the authors of those papers). As such it 
      should be seen as another work in this line of research, not a unique and 
      novel direction that it would be if it were applied using a generative 
      model that produced something closer to an actual image than geometric 
      lines. The authors point out that many other works require a good 
      amount of application-specific engineering, which they do not require. 
      While this is true, they also admit that such engineering would be a 
      useful addition. In fact, in the Introduction section they point to [17, 
      20] as examples of methods that rely on custom inference strategies -- 
      specifically, MCMC proposals that incorporate bottom-up cues -- in 
      contrast to their more general approach, and then in the Discussion 
      section they say that discriminatively trained proposals would likely be 
      important to extend their approach to more complex scenes. Novelty 
      aside, the proposed framework does have attractive properties and the 
      paper is well written, and the recent line of generative models for 
      computer vision problems seems to be worth pursuing and has yielded 
      promising results. While the probabilistic aspect is relatively 
      straightforward, it is simple and easily adjusted according to the needs 
      of the application (such as incorporating rotation), and the use of 
      Metropolis-Hastings is elegant and general. The authors report favorable 
      results on two interesting applications. It would have been better to 
      compare the system's performance on the CAPTCHAs to further 
      state-of-the-art approaches. Furthermore, while they compare directly to 
      [1] on the road-finding application, it would be helpful to provide a 
      means of comparison to the other results reported in the dataset's 
      original paper ([4]). As a technical note, it seems that the first 
      equation should contain an integral over X, which is latent and does not 
      appear on the left hand side.