Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation
Segmentation of 3D images is a fundamental problem in biomedical image analysis. Deep learning (DL) approaches have achieved state-of-the-art segmentation performance. To exploit the 3D contexts using neural networks, known DL segmentation methods, including 3D convolution, 2D convolution on planes orthogonal to 2D image slices, and LSTM in multiple directions, all suffer incompatibility with the highly anisotropic dimensions in common 3D biomedical images. In this paper, we propose a new DL framework for 3D image segmentation, based on a combination of a fully convolutional network (FCN) and a recurrent neural network (RNN), which are responsible for exploiting the intra-slice and inter-slice contexts, respectively. To our best knowledge, this is the first DL framework for 3D image segmentation that explicitly leverages 3D image anisotropism. Evaluating using a dataset from the ISBI Neuronal Structure Segmentation Challenge and in-house image stacks for 3D fungus segmentation, our approach achieves promising results comparing to the known DL-based 3D segmentation approaches.
This paper proposes a new model for biomedical image segmentation, which extends existing works in two distinct way: First, it generalizes the well-known U-Net architecture to a multi-scale setting (called kU-net), while remaining fundamentally within a 2D fully-convolutional scheme. Second, it combines several such kU-Net layers into a bi-directional convolutional LSTM (called BDC-LSTM), which can account for context given by the third dimension, while elegantly handling the common anisotropy of biomedical stacks (wherein the z-dimension is sampled at a lower spatial resolution than the xy-plane).
The paper is generally well written and easy to understand.  I quite like the proposed model: kU-net provides an answer to the ability to capture multi-scale features within a medical image, and the bi-directional LSTM scheme is an elegant way to account for broader context from the z-dimention.  However, I offer a few reservations to the paper as it currently stands.  Standard ways of dealing with anisotropy include resampling (e.g. trilinear) to isotropic voxels, or of using non-cubic kernels in fully convolutional 3D models [1,2]. For datasets in which the across-plane resolution is reasonably close to the within-plane one (e.g. fungus), this standard preprocessing would have been expected as a baseline. For models in which the across-plane resolution is much less than the within-plane one (e.g. Neuron), one can expect the impact of across-plane z-context to be small compared to the within-plane influence. It may be for this reason that the performance improvements of using RNNs for this dataset are so small. This might warrant a more extended discussion.  Much case is made of the fact that the model can handle anisotropic images; however, the model as presented can only deal with training and testing images that all exhibit the same sampling resolution. The real problem with anisotropy is that datasets are acquired on heterogenous equipment (e.g. scanners from different manufacturers) such that all images are collected at different resolution. As noted, current practice is to carry out a crude but effective resampling to a common resolution; however, it would be really nice to be able to have the model directly handle this heterogeneity.  The model’s multi-scale ability arise from a single downsampling (by a factor of 2) in the kU-net layer, with the concatenation of features at the two levels. Much simpler yet effective mechanisms have been proposed (e.g. [3]), and it would have been nice to understand whether they could serve the same purpose within the studied context.  The experimental results appear fairly limited. Although the proposed models seem to show improvements to commonly-reported metrics, the evaluation is limited to two very small datasets, and give no hint at how the model would perform on more general tasks (e.g. on anatomical data such as BRATS).  Moreover, the improvements appear relatively small (especially for Neuron dataset) given the enormous complexity of the model compared to baseline U-Net (both in terms of architecture and number of parameters). This at least warrants a discussion of whether the additional effort is really worth it.  Detailed Comments: - Line 85: this arrangement is also similar to [3] - Line 164-165: the argument that BDC-LSTM is more efficient at exploiting inter-splice contexts than Pyramid-LSTM needs to be better explained and substantiated. - Line 186: define deconvolution: is it simple upsampling or a real deconvolution? - Lines 211-212: it is not clear from the paper if decoupled training is indeed used in the experiments. What are the implications for model initialization? Line 330: lstm ==> LSTM Line 332: lstm ==> LSTM  References  [1] Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., ... & Heng, P. A. (2016). Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks. IEEE transactions on medical imaging,35(5), 1182-1195.  [2] A. A. A. Setio et al. (2016). Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks. in IEEE Transactions on Medical Imaging, 35(5), pp. 1160-1169.  [3] Shen, W., Zhou, M., Yang, F., Yang, C., & Tian, J. (2015, June). Multi-scale convolutional neural networks for lung nodule classification. In Information Processing in Medical Imaging (pp. 588-599). Springer International Publishing.