For the challenging semantic image segmentation task the best performing models have traditionally combined the structured modelling capabilities of Conditional Random Fields (CRFs) with the feature extraction power of CNNs. In more recent works however, CRF post-processing has fallen out of favour. We argue that this is mainly due to the slow training and inference speeds of CRFs, as well as the difficulty of learning the internal CRF parameters. To overcome both issues we propose to add the assumption of conditional independence to the framework of fully-connected CRFs. This allows us to reformulate the inference in terms of convolutions, which can be implemented highly efficiently on GPUs .Doing so speeds up inference and training by two orders of magnitude. All parameters of the convolutional CRFs can easily be optimized using backpropagation. Towards the goal of facilitating further CRF research we have made our implementations publicly available.
The authors propose an efficient method to perform message passing on a truncated Gaussian kernel CRF. The main contributions are the definition of a specific form of truncated Gaussian kernel that allows for fast message passing via convolutions, and the implementation of such parallelized message passing on GPU. 
In my opinion, the paper fails to convey the main idea in a clear and precise manner, the notation is mixed and often confusing, furthermore there are a number of sentences that should be rephrased to be less sensationalist, or removed. The experiments seem to show performance in par with the FullCRF on decoupled training, which seem in contrast with the much bigger performance gain of the first experiment on syntetic data. No discussion has been provided as to the possible reasons of this performance gap, although the experimental settings appear to be similar. Finally, in the last experiments with end-to-end training the authors report a performance improvement over CRFasRNN, a 3 years old paper that is far away in terms of performance with the current SOTA on Pascal VOC. The authors base on a different network than that of the CRFasRNN baseline (i.e., the difference is not only in the CRF implementation, but rather the whole network before the CRF in the proposed method), it is therefore difficult to say whether the performance improvement is due to the ResNet101 + FCN unary potentials, which is not a contribution of this manuscript, or to the proposed CRF. In general, I believe that the considerable speed gain of the proposed method might be enough to justify a publication, but the paper should be phrased in that sense if that was the intention of the authors. It is unclear to me whether the main contribution they claim is segmentation performance (IoU) or speed or both. The main contributions of this work should be stated clearly, and the modelling differences w.r.t. the FullCRF model that they aim to improve should be more explicit in the text rather than let to the reader to infer comparing the formulas.
On these grounds, I suggest a major revision of the paper and I don't recommend publication at this stage.
MAJOR
1) I firmly advocate against making strong claims, unless supported by solid proofs. I strongly recommend to rephrase, if not remove, exaggerate claims such as:
a) "[deep networks] lack the capability to utilize context information and cannot model interactions between predictions directly". This is simply not true. Any CNN with enough layers will exploit contextual information. Furthermore, any autoregressive model will model the interaction between predictions directly. See e.g., "RiFCN: Recurrent Network in Fully Convolutional Network for Semantic Segmentation of High Resolution Remote Sensing Images" by Mou et Al., "ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation" by Visin et Al., or "Predicting Deeper into the Future of Semantic Segmentation" by Luc et Al. for video semantic segmentation.
b) "CRF inference is two orders of magnitude slower than CNN inference": this, of course, depends on the kind of CRF.
c) "The long training times of the current generation of CRFs also make more in-depth research and experiments with such structured models impractical": again, not true. While it's true that CRFs tend to be slow, research with such models is not impractical and indeed there are papers that focus exactly on that (among the others, some of the ones cited in this manuscript).
d) "we propose to add the strong and valid assumption of conditional independence": as with every assumption, this is an approximation. I wouldn't claim it to be valid nor invalid, as it is simply a modeling choice.
e) "Predictions are pixel-wise and conditionally independendent (given the common feature base of nearby pixels). Structured knowledge and background context is ignored in these models.". It's unclear what is meant by "structured knowledge", but to my best understanding this sentence is misleading or wrong. Background context is considered by CNN-based models, as well as the general structure in the image.
f) "In the context of semantic segmentation most CRF based approaches are based on the Fully Connected CRF". CRFs have been used much earlier than 2011, before Fully Connected CRF was published.
g) "This makes the theoretical foundation of ConvCRF very promising, strong and valid assumptions are the powerhouse of machine learning modelling." The authors here propose a logical, but quite obvious, approximation, i.e., to constrain the CRF to model dependencies in a local neighborhood. This is the same implicit assumption of many other Gaussian kernels and algorithms for approximated inference. For how it's surely valid, and possibly strong, I don't see how it can make a "theoretical foundation". The self-congratulatory closure is unnecessary, and inappropriate.
2) While CRFs have been used for a long time on visual data, the citations in this work focus mostly on the last few years. I suggest to add at least one of the following:
* “Discriminative fields for modeling spatial dependencies in natural images" 2003
* "Multiscale conditional random fields for image labelling" 2004
* “Textonboost: Joint appearance, shape and context modeling for mulit-class object recognition and segmentation,” 2006
It could also be beneficial to the reader to add to the references broad overview works, such as "An Introduction to Conditional Random Fields" by Charles Sutton and Andrew McCallum, 2011, and/or "Structured prediction and learning in computer vision" by Nowozin and Lampert, 2011.
3) Line 5 of the algorithm adds the unary potentials at each iteration of message passing. Can you elaborate on the motivation behind this choice? The algorithm is already initialized with such potentials, and to my best knowledge unary potentials are not usually added in the mean field message passing loop. It would be interesting to compare the performance of the algorithm with and without this addition.
4) Sec4.1 reports that the filters are constant over the channel dimension c and that, in other words, this can be seen as applying the convolution over the dimension c. I fail to understand this sentence. My understanding from the formula is that the same kernel is applied to all the channels of the input, i.e., the channels of the input fed to the CRF are all processed in the same way. This should be explained clearly and the reasoning behind this choice should be explained as well. Furthermore, if I am not mistaken the authors learn a different filter in each position *of the input* rather than reusing the same filters at every position. This choice should be clarified and discussed.
5) Regarding the implementation, is there a reason not to apply a convolution with a flipped kernel to compute the cross-correlation? Also, IIRC if the kernel is symmetric (as should be for a Gaussian kernel) convolution and cross-correlation are the same. 
6) The notation is often ambiguous and at times unnecessarely heavy. I strongly recommend to go over the manuscript and use a consistent notation, making sure that every element of the notation is introduced before or right after it's used. In particular,
a) Sec3: in the text, a segmentation instance is referred to as X, while in the formula as \hat x. \tilde I is never introduced. k_{\alpha} is defined but I believe never used (I suggest to drop the name if there is no reference to it). Is there a reason to drop the subscript G in the FullCRF pairwise potential? Or conversely, is there a reason to have it everywhere else? Furthermore, there doesn't seem to be a difference between k_G, k_g and g, I suggest to use only one consistent notation to refer to the Gaussian kernels throughout the paper. It's also unclear if the I superscript is needed for the feature vectors. 
b) Sec4.1, The shape of the Gaussian kernels is the same as that of the input. I believe that the input in this context refers to a patch and not to the whole image. If so, this should be specified, otherwise the dimensions of the kernel should be referred to with a different letter than those of the input. 
c) Sec4.1, I believe dx and dy refer to the in-kernel displacement. Their semantic is not clear from the text and should be defined properly.
d) Sec4.1, the feature vectors are defined in the text as f_1..f_d. The formula of the kernel uses f_i^{(d)} instead. It's unclear what the superscripts stands for and whether it is actually useful or redundant.
e) Sec4.1, x and y are not defined, I suspect they refer to the position of the pixel, which was previously encoded as p_i and p_j. Once again, the notation should be consistent across the manuscript.
f) In the definition of the convCRFs, w is used for the width of the input, w_i for the weights of the kernels. In the FullCRF, w^{(1)} and w^{(2)} for the potentials, w^{(m)} for the sum over the kernels. k is used for the kernel dimension, k_G to refer to the kernel itself, as well as g. The notation could be made less ambiguous and consistent (superscript vs subscript semantics).
g) Sec3, the number of pixel is defined as n but in Formula 2 N is used instead.
g) Vectors and matrices should be bold-face. The use of capital letters for constants might also improve the readability of the manuscript.
7) The experiments with the Conv (ConvCRF?) variants of Table 2 are not discussed in the text.
8) Although Sec5.2 concludes with "The experiments also confirm the observation of Sec5.1, that ConvCRF performs slightly better than FullCRF", Sec5.1 reported that "it can be seen that ConvCRFs outperform FullCRFs significantly". The authors should decide whether the results are slightly better or outperform the baseline. In general a in-depth discussion on the performance of the algorithm is missing.
9) Sec5.3, it's unclear what this sentence means "introduce an auxiliary unary loss to counterbalance the vanishing gradient problem". If such a term has been added, it should be reported in a formula and it's effectiveness should be supported by experimental data.
MINOR
m1) In the related work, the sentence "transposed convolution layers are applied at the end of the prediction pipeline ot produce high-resolution output" seems to suggest these are always applied, while many recent methods rely on bilinear upsampling to recover the original resolution. Please rephrase it accordingly.
m2) In Parameter learning in CRF: "the idea utilizes, that for the message passing the identity .. is valid." This sentence doesn't make any sense to me. Is it possible it is a leftover?
m4) In Sec3, the features vectors [...] may depend on the input image I. I am confused as to when they might be independent of the image. Can you elaborate on that?
m5) In Sec3, it's unclear to me what the vertical bar in the Pot model stands for. I believe the correct formula should be 1_{[xi != xj]}.
m6) In mean field inference, Algorithm 1 does not refer to FullCRFs.
m7) End of page 6, "Note that this gives FullCRFs a natural advantage. The performance of CRFs however is very robust [...]". Why is this an advantage for FullCRFs? How does that relate to the following sentence?
m8) Sec4.1, the authors claim that one of the key contribution of the paper is that exact message passing is efficient. Given the locality assumption, message passing is approximate - which is also why it's efficient. The authors could instead argue that using convolutions is faster and possibly leads to better final performance than using the permutohedral lattice approximation (although it's unclear whether this is the case from the experiments), with proper reference to compelling results in this direction.
Finally, a few typos:
* Abstract, space missing after GPUs
* Introduction, Convolutional Neuronal -> Convolutional Neural
* Introduction, order of magnitude slower then -> than
* Introduction, to slow -> too slow
* Parameter learning in CRF: missing space before proposed to use gradient descent
* Parameter learning in CRF: gradient decent -> descent
* Parameter learning in CRF: extra comma after "another advantage of this method is"
* Sec 3: "weighted sum of Gaussian kernels", the apex of the second should be "M" I believe.
* Sec 3: "can be chosen arbitrary" -> arbitrarily
* Sec 5.2, beginning of page 8: then -> than