We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts. The network learns to extract the most significant K patches, and feeds these patches to a task-specific networke.g., auto-encoder or classifier -to solve a domain specific problem. The challenge in training such a network is the non-differentiable top-K selection process. To address this issue, we lift the training optimization problem by treating the result of top-K selection as a slack variable, resulting in a simple, yet effective, multi-stage training. Our method is able to learn to detect recurring structures in the training dataset by learning to reconstruct images. It can also learn to localize structures when only knowledge on the occurrence of the object is provided, and in doing so it outperforms the state-of-the-art.
The article introduces the novel MIST architecture which tries to solve the problem of multiple-instance classification and image generation from multiple objects. It employs two submodels, where the first generates a heatmap of intereting region and the second model is a task-specific model that works on image-patches, for example a classifier or an autoencoder. Both models are connected by a patch-extraction routine. The main contribution of this paper is to provide a way to propagate errors through this non-differentiable patch-extraction scheme. This is done by introducing slack-variables.
------------------
The paper is overall relatively easy to follow and the results are very good. However, it suffers from the fact that it does not differentiate between model-architecture and the overall approach. While the main contribution is described in Section 4, the paper spends a lot of space beforehand to introduce the task-dependent models as well as the heatmap architecture - things that i can imagine will vary a lot in different applications. The real important part is how to train the model and this is unfortunately only half described. A good deal of abstraction from the network architecture would have made the paper a lot better.  Further, I think that the loss-function for the classification task does not work in the general case.
On my first read-through, i completely misunderstood Section 4. Here is an unsorted list of issues i had with this:
- since E_K is not truly invertible, writing the approximate inverse as E_K^{-1} is misleading. 
- It might help to stress that you treat {x_k} as continuous and the sampling as differentiable.
- In (7) it would be better to explicitly write E_K^{-1}(x_k) instead of introducing \bar{h}. The line below is not clear.
- It is also misleading, because the choice of \bar{h}=E_K^{-1}(x_k) is not the minimizer of (5) given that all other variables are fixed. You can see this by observing that assuming that when {x_k}=E_K(H(I)) holds, we can choose all other pixels
to be exactly the value returned by H(I).
- I am not sure where the alternating part comes from because this usually involves taking your solution from (7) and feeding it into (6). 
- I am pretty sure that in (6)+(7), as well as lines 3+6 of the algorithm, you actually don't want to optimize for tau or eta from scratch but only perform a single SGD step. I think that is what you are doing, but right now it is written as "find a complete new model for each batch".
- Since this is performed batch-wise: is x_k a variable kept between iterations or do you use H(I) for an initial estimate of x_k for the batch?
Regarding the classification objective:
-since (3) uses the MSE of the mean class-label and the mean-prediction, a dataset where all objects always appear with the exact same amount will not work since than for each image the mean label is identical.
- Therefore, the MNIST-easy dataset should be unsolvable for the proposed architecture since every digit occurs exactly once.