Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images
Proteins are responsible for most of the functions in life, and thus are the central focus of many areas of biomedicine. Protein structure is strongly related to protein function, but is difficult to elucidate experimentally, therefore computational structure prediction is a crucial task on the way to solve many biological questions. A contact map is a compact representation of the three-dimensional structure of a protein via the pairwise contacts between the amino acids constituting the protein.
We use a convolutional network to calculate protein contact maps from detailed evolutionary coupling statistics between positions in the protein sequence. The input to the network has an image-like structure amenable to convolutions, but every "pixel" instead of color channels contains a bipartite undirected edge-weighted graph. We propose several methods for treating such "graph-valued images" in a convolutional network. The proposed method outperforms state-of-the-art methods by a considerable margin.
Protein structure is mediated and stabilized by series of weak interactions (physical contacts) between pairs of its amino acids. Let L be the length of the sequence of a protein (i.e. the number of its amino acids). The tertiary structure can be partially summarized as a so-called contact map -a sparse L × L matrix C encoding the presence or absence of physical contact between all pairs of L amino acid residues of a protein. The entry C i,j is equal to 1 if residues i and j are in contact and 0 if they are not. Intermediate values may encode different levels of contact likeliness.
We use these intermediate values without rounding where possible because they hold additional information. The "contact likeliness" is a knowledge-based function derived from Protein Data Bank, dependent on the distance between Cβ atoms of involved amino acids and their type. It has been parametrized based on the amino acids' heavy atoms making biophysically feasible contact in experimentally determined structures.
A novel application of convolutional neural networks to protein co-evolution matrixes is presented that performs significantly better than state of the art approaches for evolutionary conservation based contact prediction. This method has the potential to directly improve ab initio structure prediction, which is an important and unsolved problem in biophysics and machine learning. 
Overall the paper presents a novel application of CNNs to co-evolution Protein Contact Prediction (PCP) with state-of-the-art results. The paper is a strong contribution to the field of coevolution PCP. The method has the potential to improve ab initio structure prediction accuracy, which could have wide ranging impacts on studying proteins without solved structural complexes.  The paper itself contains flaws in the experimental setup, how the methods are described, and how the results are presented.  Key issues: - Experimental setup: not enough unique training examples are used to compare to existing methods. More than 100K examples exist for this specific task, but only 307 are used. - A key related method (PconsC2) is missing from the experimental benchmarks. Method description: the model is not described in sufficient detail to be reproduced by a 3rd party with a reasonable level expertise. In particular, the loss function and model parameters are not sufficiently described. - Results presentation and interpretation: plots are hard to read because underlying data points are too lightly printed, too few examples are used for results comparisons, and several key claims are not sufficiently supported by data.  Specific points  - The number of examples used for training and testing is very small, especially when compared to numbers of examples used in related publications. For example, in the plmDCA paper, 10,000 examples are used, rather than the 307 in this paper. The authors claim they had trouble finding consistent homologue sets for all proteins, but there are exisiting pipelines for this task (e.g. the gremlin pipeline http://gremlin.bakerlab.org/). It would be especially interesting to see how the relative performance of the method changes when trained on a larger corpus of training examples, especially considering that PDB contains > 100K unique crystal structures that could each potentially be used as an example.  - No justification is given for the binary contact/non-contact threshold of 8A (previous comparisons have used different cutoffs). It is likely that cutoffs at different resolutions will be useful for ab initio structure prediction. It would be interesting to see a comparison of pmlConv to plmDCA and CONSIP2 with different cutoff boundaries. It is potentially a concern that the cutoff was chosen because it maximizes the margin of plmConv over other methods, rather than because it corresponds to a physically important distance.  - The model should also be benchmarked against PconsC2 (Skwark M., et al., “Improved Contact Predictions Using the Recognition of Protein Like Contact Patterns”, PLOS Computational Biology, 2014). This would be an especially informative comparison because PconsC2 uses an ensemble of linear classifiers for the exact same task (PCP from plmDCA matrixes). PconsC2 also includes learned spatial filters, which are missing from the current benchmarks.  - The deep learning model is not adequately described to be reproducible. Especially important is that the paper does not mention what loss function is used for optimization (presumably binary cross entropy?) Also missing are descriptions of what convolutional strides and padding is used, what types of initialization are used (probably very important for learning on limited examples), whether batch norm is used, and whether dropout is used.  - No interpretation is presented of what features the model learns. It would be interesting to apply a technique such as guided back propagation to input to provide an interpretation of which features are most important for classification of a given residue pair.  - In figures 2 and 3, it is very hard to see the circles that represent individual data points. The circles should be presented with higher contrast so readers can better see the underlying data distribution, especially given that it is so sparse.  - In figure 3, the presented data are very noisy. It would be worth reproducing this figure with more alignments across a larger set of proteins.  - In figure 3, the x-axis is labeled “amount of sequences”, but the caption states the figure shows the “information content” of the alignment as the number of “non-redundant sequences”. It would be worthwhile to consider more informative metrics than the number of non-redundant sequences: an alignment in which each sequence has hamming distance exactly 1 from each other sequence is much less informative for co-evolution than an alignment with higher sequence divergence. A metric such as effective number of sequences (where sequences are weighted by normalized average hamming distance), or alignment entropy may be more informative summary statistics for this purpose.  - The paper presents a semantic argument that larger layers could lead to overfitting, but could this be overcome through parameter regularization or dropout? This would allow convolutional filters to potentially detect longer range interactions.  - The paper claims that plmConv yields higher accuracy than plmDCA for especially long range interactions. It would be interesting to plot PPV as a function of contact distance, since this figure would directly support this point (rather than the PPV table split by distance, which only partially supports this claim).  - Section 2.4 proposes 4 schemes for featurizing the J matrix, but results are not reported for each. It would be informative to report how featurization scheme influences performance.