Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin
The past decade has seen a revolution in genomic technologies that enabled a flood of genome-wide profiling of chromatin marks. Recent literature tried to understand gene regulation by predicting gene expression from large-scale chromatin measurements. Two fundamental challenges exist for such learning tasks: (1) genome-wide chromatin signals are spatially structured, high-dimensional and highly modular; and (2) the core aim is to understand what the relevant factors are and how they work together. Previous studies either failed to model complex dependencies among input signals or relied on separate feature analysis to explain the decisions. This paper presents an attention-based deep learning approach, AttentiveChrome, that uses a unified architecture to model and to interpret dependencies among chromatin factors for controlling gene regulation. AttentiveChrome uses a hierarchy of multiple Long Short-Term Memory (LSTM) modules to encode the input signals and to model how various chromatin marks cooperate automatically. AttentiveChrome trains two levels of attention jointly with the target prediction, enabling it to attend differentially to relevant marks and to locate important positions per mark. We evaluate the model across 56 different cell types (tasks) in humans. Not only is the proposed architecture more accurate, but its attention scores provide a better interpretation than state-of-the-art feature visualization methods such as saliency maps.
DNA. These spatial re-arrangements result in certain DNA regions becoming accessible or restricted and therefore affecting expressions of genes in the neighborhood region. Researchers have established the "Histone Code Hypothesis" that explores the role of histone modifications in controlling gene regulation. Unlike genetic mutations, chromatin changes such as histone modifications are potentially reversible ( [5]). This crucial difference makes the understanding of how chromatin factors determine gene regulation even more impactful because this knowledge can help developing drugs targeting genetic diseases.
At the whole genome level, researchers are trying to chart the locations and intensities of all the chemical modifications, referred to as marks, over the chromatin 4 . Recent advances in nextgeneration sequencing have allowed biologists to profile a significant amount of gene expression and chromatin patterns as signals (or read counts) across many cell types covering the full human genome. These datasets have been made available through large-scale repositories, the latest being the Roadmap Epigenome Project (REMC, publicly available) ( [18]). REMC recently released 2,804 genome-wide datasets, among which 166 datasets are gene expression reads (RNA-Seq datasets) and the rest are signal reads of various chromatin marks across 100 different "normal" human cells/tissues [18].
The fundamental aim of processing and understanding this repository of "big" data is to understand gene regulation. For each cell type, we want to know which chromatin marks are the most important and how they work together in controlling gene expression. However, previous machine learning studies on this task either failed to model spatial dependencies among marks or required additional feature analysis to explain the predictions (Section 4). Computational tools should consider two important properties when modeling such data.
• First, signal reads for each mark are spatially structured and high-dimensional. For instance, to quantify the influence of a histone modification mark, learning methods typically need to use as input features all of the signals covering a DNA region of length 10, 000 base pair (bp) 5 centered at the transcription start site (TSS) of each gene. These signals are sequentially ordered along the genome direction. To develop "epigenetic" drugs, it is important to recognize how a chromatin mark's effect on regulation varies over different genomic locations.
• Second, various types of marks exist in human chromatin that can influence gene regulation. For example, each of the five standard histone proteins can be simultaneously modified at multiple different sites with various kinds of chemical modifications, resulting in a large number of different histone modification marks. For each mark, we build a feature vector representing its signals surrounding a gene's TSS position. When modeling genome-wide signal reads from multiple marks, learning algorithms should take into account the modular nature of such feature inputs, where each mark functions as a module. We want to understand how the interactions among these modules influence the prediction (gene expression).
In this paper we propose an attention-based deep learning model, AttentiveChrome, that learns to predict the expression of a gene from an input of histone modification signals covering the gene's neighboring DNA region. By using a hierarchy of multiple LSTM modules, AttentiveChrome can discover interactions among signals of each chromatin mark, and simultaneously learn complex dependencies among different marks. Two levels of "soft" attention mechanisms are trained, (1) to attend to the most relevant regions of a chromatin mark, and (2) to recognize and attend to the important marks. Through predicting and attending in one unified architecture, AttentiveChrome allows users to understand how chromatin marks control gene regulation in a cell. In summary, this work makes the following contributions:
• AttentiveChrome provides more accurate predictions than state-of-the-art baselines. Using datasets from REMC, we evaluate AttentiveChrome on 56 different cell types (tasks).
• We validate and compare interpretation scores using correlation to a new mark signal from REMC (not used in modeling). AttentiveChrome's attention scores provide a better interpretation than state-of-the-art methods for visualizing deep learning models. • AttentiveChrome can model highly modular inputs where each module is highly structured.
AttentiveChrome can explain its decisions by providing "what" and "where" the model has focused 4 In biology this field is called epigenetics. "Epi" in Greek means over. The epigenome in a cell is the set of chemical modifications over the chromatin that alter gene expression. on. This flexibility and interpretability make this model an ideal approach for many real-world applications.
• To the authors' best knowledge, AttentiveChrome is the first attention-based deep learning method for modeling data from molecular biology.
In the following sections, we denote vectors with bold font and matrices using capital letters. To simplify notation, we use "HM" as a short form for the term "histone modification".
The paper presents a novel method for predicting gene regulation by LSTM with an attention mechanism. The model consists of two levels, where the first level is applied on bins for each histone modifications (HM) and the second level is applied to multiple HMs. Attention mechanism is used in each level to focus on the important parts of the bins and HMs. In the experiments, the proposed method improves AUC scores over baseline models including CNN, LSTM, and CNN with an attention mechanism. This is an interesting paper which shows that LSTM with an attention mechanism can predict gene regulation. 
1. It is unclear to me why the second level is modeled using LSTM because there is no ordering between HMs. Would it be reasonable to use a fully connected layer to model dependencies between HMs? Based on Table 1 and 2, the one level model (LSTM-\alpha) outperforms the two level model ((LSTM-\alpha,\beta) in many cases. It would be interesting to investigate how HMs are coupled with each other in the learned model.
2. In Section 2, an algorithm box that includes the equations for the entire model would be very helpful. 
3. It is unclear to me if DeepChrome was compared with the proposed method in the experiments. It would be helpful to indicate which model corresponds to DeepChrome.
4. As baselines, it would be helpful to include non-neural network models such as SVM or logistic regression. 
5. To allow researchers in biology to use this method, it would be very helpful to have the source code publicly available with URLs in the paper.