Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or Deep 3D Pan, with "t-shaped" adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes and our VXXLXX STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the stateof-the-art method, SOTA, by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the "t-shaped" kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method.
Summary:
This paper proposes a deep learning method to produce "pans" of an input image. That is, simulated images of the scene from translated viewpoints. Unlike some previous work that considers only a fixed baseline (such as the 2nd view of a stereo camera), this approach allows generation of a range of views. A specially crafted convolutional architecture is shown to be well-suited to this problem. Results demonstrate visually pleasing image generation and low metric errors on several datasets.
Strengths:
- The justifications for the design choices in this paper, in particular the convolution structure and connection to image geometry, was quite strong compared to recent papers (although, many of the presented ideas are known in more classical, non-learning, techniques). 
- All presented empirical results are impressive, and show the method is likely to "really work" and be reproducible, judging from the number of experiments where the method has consistently outperformed. 
- The method is clear and straightforward to implement either on its own, or as a module/architecture within a larger pipeline.
Areas for Improvement and Detailed Suggestions:
- The problem of panned view generation is a bit more narrow than some authors are lately considering (generate any viewpoint including off-axis rotations). 
- The t-shaped network architecture here is largely presented as only appropriate to handing image panning. Could a more general network be created, perhaps parameterized by the type of rigid motion occurring? Better, could more flexible networks be proposed with sparsity constraints that allow sub-patterns like the t-network to be learned in a data-driven fashion? 
- Please try to cite referenced work more consistently. For several methods, such as Deep 3D Zoom Net, you have begun to discuss the method using name only for large stretches. It would be helpful to keep using the citation at least once per paragraph to remind the reader of the source of these ideas. 
Decision and Justification: 
Weak reject due to the lack of generality in the approach. I suspect the impact of this work will be a bit less than the very competitive bar for ICLR this year. However, I must note that I am the least expert in this area, out of any paper in my stack.