Predicting the outcome of pharmacological assays based on high-resolution microscopy images of treated cells is a crucial task in drug discovery which tremendously increases discovery rates. However, end-to-end learning on these images with convolutional neural networks (CNNs) has not been ventured for this task because it has been considered infeasible and overly complex. On the largest available public dataset, we compare several state-of-the-art CNNs trained in an end-to-end fashion with models based on a cell-centric approach involving segmentation. We found that CNNs operating on full images containing hundreds of cells perform significantly better at assay prediction than networks operating on a single-cell level. Surprisingly, we could predict 29% of the 209 pharmacological assays at high predictive performance (AUC > 0.9). We compared a novel CNN architecture called "GapNet" against four competing CNN architectures and found that it performs on par with the best methods and at the same time has the lowest training time. Our results demonstrate that end-to-end learning on high-resolution imaging data is not only possible but even outperforms cell-centric and segmentation-dependent approaches. Hence, the costly cell segmentation and feature extraction steps are not necessary, in fact they even hamper predictive performance. Our work further suggests that many pharmacological assays could be replaced by high-resolution microscopy imaging together with convolutional neural networks.
The authors explore the possibility of using an end-to-end approach for predicting pharmacological assay outcome using fluorescence microscopy images from the public Cell Painting dataset. In my view, the primary contributions are the following: an interesting and relatively new application (predicting assay outcomes), enriching the CellPainting dataset with drug activity data, and a comparison of several relevant methods and architectures. The technical novelty is weak, and although the authors demonstrate that end-to-end holistic approaches outperform previous segmentation-and-feature-extraction approaches, this result is not surprising and has been previously reported in closely related contexts.
OVERVIEW
The authors evaluate the possibility of using and end-to-end deep learning approach to predict drug activity using only image data as input. The authors repurpose the CellPainting dataset for activity prediction by adding activity data from online ChEMBL databases. If made available as promised, the dataset will be a valuable resource to the community. The authors compare a number of previous approaches and state-of-the-art image classification network architectures to evaluate the use of CNNs instead of more classical image analysis pipelines. The comparison is a strong point of the paper, although some details are lacking. For example, the authors claim that GapNet is the quickest method to train, and while they report the number of hyperparameters and time per epoch, the number of epochs trained is never mentioned. 
The authors propose an architecture (GapNet) for the assay prediction task. While the way Global Average Pooling is used to extract features at different stages in the network might be new, it is a straightforward combination of GAP and skip connections. Little insight into why this approach is more efficient or evidence for its effectiveness is provided. Similarly, more explanation for why dilated convolutions and SELU activations would be appreciated. A comparison between GapNet and the same network without the GAP connections could possibly provide a more interesting comparison and might also provide a more pervasive argument as to why GapNet’s should be used. Ultimately, the benefit of using GapNet over the other architectures is not strongly motivated, as training time is less of a concern in this application than predictive power.
RELATED WORK
The authors present previous work in a clear and comprehensive manner. However, the reported finding that “CNNs operating on full images containing hundreds of cells can perform significantly better at assay prediction than networks operating on a single-cell level” is not surprising, and partial evidence of this can be found in the literature. In [1], it was shown that penultimate feature activations from pre-trained CNNs applied to whole-image fluorescence microscopy data (MOA prediction) outperform the baseline segmentation-then-feature extraction method (FNN). Similarly, in [2] (the paper proposing MIL-Net), it is shown that end-to-end whole-image CNN learning for protein localization outperforms the baseline (FNN). In [3] whole image end-to-end learning outperforms whole image extracted features for a phenotyping task. All of these references use fluorescence microscopy data similar to the dataset in this work.
[1] Pawlowski, Nick, et al. "Automating morphological profiling with generic deep convolutional networks." bioRxiv (2016): 085118.
[2] Kraus, Oren Z., Jimmy Lei Ba, and Brendan J. Frey. "Classifying and segmenting microscopy images with deep multiple instance learning." Bioinformatics 32.12 (2016): i52-i59
[3] Godinez, William J., et al. "A multi-scale convolutional neural network for phenotyping high-content cellular images." Bioinformatics 33.13 (2017): 2010-2019.
APPROACH
The authors compile enrich the CellPaining dataset with activity data from various drug discovery assays. In my view, the creation of this dataset is the strongest and most valuable contribution of the paper. The method used to collect the data is described clearly and the choices made when compiling the dataset, including the thresholds and combinations of activity measures seems like a well founded approach.
The authors then identify a number of approaches that are relevant for the problem at hand, binary prediction of drug activity based on image data. These include previous approaches used for cell images and modern image classification networks.
EXPERIMENTS
The different approaches/networks mentioned above were evaluated on a testset. The results indicate that end-to-end CNN approaches outperform all non-end-to-end with no significant difference between the individual end-to-end CNNs. The results are stated clearly and the presentation of different metrics is a nice addition to properly compare the results. It would however contribute valuable information if the authors stated how the confidence intervals of the F1 score are calculated (are the experiments based on several runs of each network or how is it done).
NOVELTY/IMPACT
+ Creation of a new dataset on a new and interesting problem 
+ Useful comparison of modern networks on the task
- GapNet - lacking technical novelty, insight, and performance is unconvincing
- Demonstrates that end-to-end learning outperforms cell centric approach - was this really surprising or even new information?
OTHER NOTES:
* Figure 3 is never mentioned in the main text
* Figure 3 (*’s) are confusing. Do they represent outliers? Statistical significance tests?
* Figure 5 which panel is which?
* Be clear what you mean when you refer to “upper layers” of a network
* An important point not mentioned: in practice, many assays use stains that are closely tied to the readout, unlike the dataset here which provides only landmark stains. The results found here do not necessarily apply in other cases.