HUMAN-LEVEL PROTEIN LOCALIZATION WITH CON- VOLUTIONAL NEURAL NETWORKS
Localizing a specific protein in a human cell is essential for understanding cellular functions and biological processes of underlying diseases. A promising, low-cost, and time-efficient biotechnology for localizing proteins is high-throughput fluorescence microscopy imaging (HTI). This imaging technique stains the protein of interest in a cell with fluorescent antibodies and subsequently takes a microscopic image. Together with images of other stained proteins or cell organelles and the annotation by the Human Protein Atlas project, these images provide a rich source of information on the protein location which can be utilized by computational methods. It is yet unclear how precise such methods are and whether they can compete with human experts. We here focus on deep learning image analysis methods and, in particular, on Convolutional Neural Networks (CNNs) since they showed overwhelming success across different imaging tasks. We propose a novel CNN architecture "GapNet-PL" that has been designed to tackle the characteristics of HTI data and uses global averages of filters at different abstraction levels. We present the largest comparison of CNN architectures including GapNet-PL for protein localization in HTI images of human cells. GapNet-PL outperforms all other competing methods and reaches close to perfect localization in all 13 tasks with an average AUC of 98% and F1 score of 78%. On a separate test set the performance of GapNet-PL was compared with three human experts and 25 scholars. GapNet-PL achieved an accuracy of 91%, significantly (p-value 1.1e −6 ) outperforming the best human expert with an accuracy of 72%.
This manuscript describes a deep convolutional neural network for
assigning proteins to subcellular compartments on the basis of
microscopy images.
Positive points:
- This is an important, well-studied problem.
- The results appear to improve significantly on the state of the art.
- The experimental comparison is quite extensive, including
  reimplementations of four, competing state-of-the-art methods, and
  lots of details about how the comparisons were carried out.
- The manuscript also includes a human-computer competition, which the
  computer soundly wins.
- The manuscript is written very clearly.
Concerns:
There is not much here in the way of new machine learning methods.
The authors describe a particular neural network architecture
("GapNet-PL") and show empirical evidence that it performs well on a
particular dataset.  No claims are made about the generalizability of
the particular model architecture used here to other datasets or other
tasks.
A significant concern is one that is common to much of the deep
learning literature these days, namely, that the manuscript fails to
separate model development from model validation. We are told only
about the final model that the authors propose here, with no
discussion of how the model was arrived at.  The concern here is that,
in all likelihood, the authors had to try various model topologies,
training strategies, etc., before settling on this particular setup.
If all of this was done on the same train/validation/test split, then
there is a risk of overfitting.
The dataset used here is not new; it was the basis for a competition
carried out previously.  It is therefore somewhat strange that the
authors chose to report only the results from their reimplementations
of competing methods.  There is a risk that the authors'
reimplementations involve some suboptimal choices, relative to the
methods used by the originators of those methods.
Another concern is the potential circularity of the labels.  At one
point, we are told that "Most importantly, these labels have not been
derived from the given microscopy images, but from other
biotechnologies such as microarrays or from literature."  However,
earlier we are told that the labels come from "a large battery of
biotechnologies and approaches, such as microarrays, confocal
microscopy, knowledge from literature, bioinformatics predictions and
additional experimental evidence, such as western blots, or small
interfering RNA knockdowns."  The concern is that, to the extent that
the labels are due to bioinformatics predictions, then we may simply
be learning to re-create some other image processing tool.
The manuscript contains a fair amount of biology jargon (western
blots, small interfering RNA knockdowns, antibodies, Hoechst staining,
etc.) that will not be understandable to a typical ICLR reader.
At the end, I think it would be instructive to show some examples
where the human expert and the network disagreed.
Minor:
p. 2: "automatic detection of malaria" -- from images of what?
p. 2: Put a semicolon before "however" and a comma after.
p. 2: Change "Linear Discriminant" to "linear discriminant." Also, remove
the abbreviations (SVM and LDA), since they are never used again in
this manuscript.
p. 5: Delete comma in "assumption, that."
p. 8: "nearly perfect" -> "nearly perfectly"
The confusion matrices in Figure 5 should not be row normalized --
just report raw counts.  Also, it would be better to order the classes
so that confusable ones are nearby in the list.