Documents as multiple overlapping windows into a grid of counts
In text analysis documents are often represented as disorganized bags of words; models of such count features are typically based on mixing a small number of topics [1,2]. Recently, it has been observed that for many text corpora documents evolve into one another in a smooth way, with some features dropping and new ones being introduced. The counting grid [3] models this spatial metaphor literally: it is a grid of word distributions learned in such a way that a document's own distribution of features can be modeled as the sum of the histograms found in a window into the grid. The major drawback of this method is that it is essentially a mixture and all the content must be generated by a single contiguous area on the grid. This may be problematic especially for lower dimensional grids. In this paper, we overcome this issue by introducing the Componential Counting Grid which brings the componential nature of topic models to the basic counting grid. We evaluated our approach on document classification and multimodal retrieval obtaining state of the art results on standard benchmarks.
The paper presents an LDA-style mixed membership model 
      where the component distributions are counting grids. Each document is a 
      Dirichlet mixture over windows into the grid. Each window is a uniform 
      distribution over the cells contained in that window. Each cell in the 
      grid is a distribution over the vocabulary. The method is evaluated using 
      classification accuracy, embedding quality, AUC in a retrieval task, and a 
      visualization example. I like this paper a lot. I think it's 
      marginally above the accept threshold as is, and could be substantially 
      improved with another round of revisions. I gave it an "incremental", but 
      it's more like a 1.499. The introduction does a reasonably good 
      job presenting the basic intuition of the paper, that words often appear 
      in a smooth continuum rather than distinct topics, and that the 
      single-window model is too restrictive for real documents. Figure 1 is a 
      bit of a missed opportunity. 1a works for me, but I found 1b and 1c 
      confusing. A simpler toy example might be better, especially if it could 
      be merged with Figure 2b. I had a hard time moving from the 
      introduction to section 2. In part, I felt there was a lot of technical 
      terminology floating around, some of it redundant. What is a source, for 
      example, or a bag? Are these cells and documents? I would pick a small set 
      of concrete terms and use them consistently in both sections. Avoid 
      "topic". In Sect 2: I would call pi_i(z) a distribution 
      over the vocabulary, not a set of normalized counts. The latter may be its 
      actual implementation in an algorithm, but here we're defining the model 
      in the abstract. Give an example: pi_{1,1}("pizza") is the probability of 
      the word "pizza" in the top left cell (or similar, if that's not correct) 
      What's a sample? Is that a document? What is Z? The size 
      of the vocabulary or the size of the grid? The variable z means "hidden 
      variable" to me. I'm really looking for a toy example. 
      Give an example for W, like 3x3. Actually, I might 
      restrict the initial presentation to square windows in 2D grids, for the 
      sake of simplicity of notation, and then comment that the model is easily 
      extended to d-dimensional grids and arbitrary window sizes. I 
      don't see why the number of non-overlapping windows is equivalent to the 
      number of topics. The whole point is that the windows can overlap, no? 
      Overlapping windows don't represent independent topics, but they do add 
      modeling power, and that has to be taken into account for a fair 
      comparison. Also, give a formula for kappa explicitly: it's not 
      unambiguous right now (which ratio?). I would rewrite section 2 
      around a step-by-step generative model. To generate a document, you first 
      pick a distribution over windows. Then to generate each word, you pick a 
      window, then pick a cell within the window, and finally pick a word from 
      the cell. The connection to LDA (windows of size 1x1) would be 
      good to point out here. It took a moment to figure out that U^W 
      was a uniform distribution over a window. It looked like a matrix 
      multiplication at first. "Locations in the grid generate similar 
      features" -- does this mean that the words in neighboring cells are 
      semantically similar (but the words themselves differ), or that the 
      distributions themselves are similar? "The number of sources must 
      not be specified..." First, I think this should be "need not be" -- "must 
      not be" implies that you will get in trouble if you do! Second, I'm not 
      sure what this means. The rest of the paper seems to assume that all 
      dimensions are known in advance. Also, I'm still not sure what sources are 
      (cells? windows?). I didn't follow the marginalization step. I'd 
      like a bit more explanation. If space is needed, variational update 
      equations are an excellent candidate for supplementary material. 
      Distinguish between variables and distributions, and avoid 
      referring to symbol alone: "the new updates for the cell distributions 
      q(k) and the window distributions q(l) ..." The comparison to 
      Dunson et al is reasonable, but it might be worth talking about what is 
      lost by using a grid rather than a continuous space. Section 2.1 
      seems like a tangent. Evaluation: I didn't see held-out 
      likelihood tests. It's not trivial for LDA. How would they be computed for 
      this model? I'd like to see a linear SVM used for document 
      classification, in addition to the nearest neighbor method (I'm assuming 
      that's BOW?). See my earlier concerns about using "capacity" as a 
      measure of approximate # of topics. Why not just the number of grid cells? 
      In some sense CCG is just a fancier correlated distribution over the 
      cells, like CTM or PAM. The Euclidean embedding section is 
      interesting. I'd like to see more of it. What are the visual 
      words? Are they given with the data set? On the whole I buy the 
      visualization aspect, because the word distributions are intrinsically 
      embedded in a 2D space. I'd like to see more development of this idea. How 
      would people use this embedding to learn something about the corpus? I 
      didn't find Figure 5 particularly compelling for the fry/deep fry 
      distinction, but the cluster of words that I associate with Indian recipes 
      in the bottom right was interesting.