In this paper, we propose a capsule-based neural network model to solve the semantic segmentation problem. By taking advantage of the extractable part-whole dependencies available in capsule layers, we derive the probabilities of the class labels for individual capsules through a recursive, layer-by-layer procedure. We model this procedure as a traceback pipeline and take it as a central piece to build an end-to-end segmentation network. Under the proposed framework, image-level class labels and object boundaries are jointly sought in an explicit manner, which poses a significant advantage over the state-of-the-art fully convolutional network (FCN) solutions. Experiments conducted on modified MNIST and neuroimages demonstrate that our model considerably enhance the segmentation performance compared to the leading FCN variant.
Based on the CapsNet concept of Sabour the authors proposed a trace-back method to perform a semantic segmentation in parallel to classification. The method is evaluate on MNIST and the Hippocampus dataset.
The paper is well-written and well-explained. Nevertheless, I think it would be useful to have some illustrations about the network architecture. Some stuff which is explained in text could be easily visualized in a flow chart. For example, the baseline architecture and your Tr-CapsNet could be easily explained via a flow chart. With the text only, it is hard to follow. Please think about some plots in the final version or in the appendix. One question which is aligned to that: How many convolutional filters are used in the baseline model?
Additionally, think about a pseudo-code for improved understandability. 
Some minor concerns/ notes to the authors:
1.	At page 5: You mentioned that the parameters lambda1 and lambda 2 are important hyper-parameters to tune. But in the results you are not explaining how the parameters were tuned. So my question is: How do you tuned the parameters? In which range do you varied the parameters?
2.	Page 6; baseline model: Why do you removed the pooling layers?
3.	I’m curious about the number of parameters in each model. To have a valid discussion about your model is better than the U-Net-6 architecture, I would take into account the number of parameters. In case that your model is noticeably greater, it could be that your increased performance is just due to more parameters. As long as your discussion is without the number of parameters I’m not convinced that your model is better. A comparison between models should be always fair if two models are architectural similar.
4.	Why is the magnitude of lambda1 so different between the two dataset that you used?
5.	Could you add the inference times to your tables and discuss that in addition?
6.	What kind of noise is added to MNIST?
7.	What is the state-of-the-art performance on the Hippocampus dataset?
8.	What would be the performance in your experiments with a MaskRCNN segmentation network?
9.	I’m not familiar with the Hippocampus dataset. I missed a reference where the data is available or some explaining illustrations. 
10.	For both datasets, more illustrations about the segmentation performance would be fine to evaluate your method. At least in the appendix…
	
My major concern is that both datasets are not dealing with real background noise. I’m concerned that the results are not transferable to other datasets and that the method shines promising just because of the simple datasets only. For example, due to the black background MNIST digits are well separated (if we skip that you added some kind of noise). So, from that point of view your results are not convincing and the discussion of your results appearing sparse and not complete.
To make your results transparent you could think about to publish the code somewhere.