Current state-of-the-art methods for semantic segmentation use deep neural networks to learn the segmentation mask from the input image signal as an imageto-image mapping. While these methods effectively exploit global image context, the learning and computational complexities are high. We propose shared memory augmented neural network actors as a dynamically scalable alternative. Based on a decomposition of the image into a sequence of local patches, we train such actors to sequentially segment each patch. To further increase the robustness and better capture shape priors, an external memory module is shared between different actors, providing an implicit mechanism for image information exchange. Finally, the patch-wise predictions are aggregated to a complete segmentation mask. We demonstrate the benefits of the new paradigm on a challenging lung segmentation problem based on chest X-Ray images, as well as on two synthetic tasks based on the MNIST dataset. On the X-Ray data, our method achieves state-of-the-art accuracy with a significantly reduced model size compared to reference methods. In addition, we reduce the number of failure cases by at least half.
The authors applied the external memory module proposed by Graves et al. (2016) to the image segmentation task. SHAMANN is an extension to allow memory sharing between directions. 
Authors claimed that one of the contributions is a reformulation of the semantic segmentation problem as a sequence learning task.
There are many previous works done in this direction,
- "Multi-Dimensional Recurrent Neural Networks", 2007
- "Scene Labeling with LSTM Recurrent Neural Networks", 2015
- "ReSeg: A Recurrent Neural Network-Based Model for Semantic Segmentation", 2016
- "Robust, Simple Page Segmentation Using Hybrid Convolutional MDLSTM Networks", 2017 
and many more.
Authors should compare with those LSTM-based image segmentation approaches as well.
Their second contribution is a network with a shared external memory module between directions. However, the experiments are not enough to show the benefits of it. See the details below.
Handling long-range dependencies:
- In Section 3.3.2, authors mentioned that "One limitation of Bi-LSTM is that the number of network parameters grows proportionally to the memorization capacity, making it unsuitable for sequences with long-range dependencies.". 
However, the experiments are not with long range sequences: 169 sequence length for X-ray dataset and 49 length for MNIST. A classic LSTM (not bi-directional) is known to handle up to 200 timesteps. Some comparison/analysis of handling long-range dependencies of Bi-LSTM, Bi-MANN, and SHAMANN are needed (ideally on high-resolution real images).
Dataset:
-  Authors compared 3 models only on MNIST. The structure on MNIST is simple, and the resolution of images is small to show the benefit of using (shared) external memory module instead of individual memory cells. It is not surprising that the reported performance difference is small. Authors could have reported such a comparison on X-ray dataset too but they did not. I would recommend authors pick another high-resolution real-image dataset and compare the performance of these 3 models.  
Additional comparisons:
- Various patch size
- Longer sequence length
- Especially a trade-off between the patch size and the sequence length on the high resolution images (larger patch size with a shorter sequence length or shorter patch size with a longer sequence length)
- A comparison of Bi-LSTM with sharing weights will also be a good baseline.