Efficient Approximation Algorithms for String Kernel Based Sequence Classification
Sequence classification algorithms, such as SVM, require a definition of distance (similarity) measure between two sequences. A commonly used notion of similarity is the number of matches between k-mers (k-length subsequences) in the two sequences. Extending this definition, by considering two k-mers to match if their distance is at most m, yields better classification performance. This, however, makes the problem computationally much more complex. Known algorithms to compute this similarity have computational complexity that render them applicable only for small values of k and m. In this work, we develop novel techniques to efficiently and accurately estimate the pairwise similarity score, which enables us to use much larger values of k and m, and get higher predictive accuracy. This opens up a broad avenue of applying this classification approach to audio, images, and text sequences. Our algorithm achieves excellent approximation performance with theoretical guarantees. In the process we solve an open combinatorial problem, which was posed as a major hindrance to the scalability of existing solutions. We give analytical bounds on quality and runtime of our algorithm and report its empirical performance on real world biological and music sequences datasets.
acids) and predicting proteins fold (functional three dimensional structure) are essential tasks in bioinformatics. Sequence classification algorithms have been applied to both of these problems with great success [3,10,13,18,19,20,25]. Music data, a real valued signal when discretized using vector quantization of MFCC features is another flavor of sequential data [26]. Sequence classification has been used for recognizing genres of music sequences with no annotation and identifying artists from albums [12,13,14]. Text documents can also be considered as sequences of words from a language lexicon. Categorizing texts into classes based on their topics is another application domain of sequence classification [11,15].
While general purpose classification methods may be applicable to sequence classification, huge lengths of sequences, large alphabet sizes, and large scale datasets prove to be rather challenging for such techniques. Furthermore, we cannot directly apply classification algorithms devised for vectors in metric spaces because in almost all practical scenarios sequences have varying lengths unless some mapping is done beforehand. In one of the more successful approaches, the variable-length sequences are represented as fixed dimensional feature vectors. A feature vector typically is the spectra (counts) of all k-length substrings (k-mers) present exactly [18] or inexactly (with up to m mismatches) [19] within a sequence. A kernel function is then defined that takes as input a pair of feature vectors and returns a real-valued similarity score between the pair (typically inner-product of the respective spectra's). The matrix of pairwise similarity scores (the kernel matrix) thus computed is used as input to a standard support vector machine (SVM) [5,27] classifier resulting in excellent classification performance in many applications [19]. In this setting k (the length of substrings used as bases of feature map) and m (the mismatch parameter) are independent variables directly related to classification accuracy and time complexity of the algorithm. It has been established that using larger values of k and m improve classification performance [11,13]. On the other hand, the runtime of kernel computation by the efficient trie-based algorithm [19,24] is O(k m+1 |Σ| m (|X| + |Y |)) for two sequences X and Y over alphabet Σ.
Computation of mismatch kernel between two sequences X and Y reduces to the following two problems. i) Given two k-mers α and β that are at Hamming distance d from each other, determine the size of intersection of m-mismatch neighborhoods of α and β (k-mers that are at distance at most m from both of them). ii) For 0 ≤ d ≤ min{2m, k} determine the number of pairs of k-mers (α, β) ∈ X × Y such that Hamming distance between α and β is d. In the best known algorithm [13] the former problem is addressed by precomputing the intersection size in constant time for m ≤ 2 only. While a sorting and enumeration based technique is proposed for the latter problem that has computational complexity O(2 k (|X| + |Y |), which makes it applicable for moderately large values of k (of course limited to m ≤ 2 only).
In this paper, we completely resolve the combinatorial problem (problem i) for all values of m. We prove a closed form expression for the size of intersection of m-mismatch neighborhoods that lets us precompute these values in O(m 3 ) time (independent of |Σ|, k, lengths and number of sequences). For the latter problem we devise an efficient approximation scheme inspired by the theory of locality sensitive hashing to accurately estimate the number of k-mer pairs between the two sequences that are at distance d. Combining the above two we design a polynomial time approximation algorithm for kernel computation. We provide probabilistic guarantees on the quality of our algorithm and analytical bounds on its runtime. Furthermore, we test our algorithm on several real world datasets with large values of k and m to demonstrate that we achieve excellent predictive performance. Note that string kernel based sequence classification was previously not feasible for this range of parameters.
The authors describe an approximation algorithm for k-mer (with mismatches) based string kernels.
The contribution is centered around a closed form expression of the intersection size of mismatching neighbourhoods.
The algorithm is evaluated in the context of sequence classification using SVM.
I think this is a great paper: clear motivation, good introduction, clear contribution, theoretical back-up, nice experimental results.
I have a few concerns regarding presentation and structuring, as well as doubts on relevance on the presented theory.
Presentation.
I think Section 3 is too technical. It contains a lot of notation, and a lot of clutter that actually hinder understanding the main ideas. On the other hand, intuition on crucial ideas is missing, so I suggest to move a few details into Appendix and rather elaborate high-level ideas.
-A table with all relevant quantities (and explanations) would make it much easier for a non-string-kernel person to follow establishing Theorem 3.3. An intuitive description of the theorem (and proof idea) would also help.
-Algorithm 1 is not really referenced from the main test. It first appears in Thm 3.10. It is also very sparsely annotated/explained. For example, the parameter meanings are not explained. Subroutines also need annotation. I think further deserves to be in the center of Section 3's text, and the text should describe the high level ideas.
-It should be pointed out more that Thm 3.3 is the open combinatoric problem mentioned in the abstract.
-In the interest of readability, I think the authors should comment more on what it is that their algorithm approximates, what the parameters that control the approximation quality are, and what would be possible failure cases.
-Theorem 3.13 should stay in the main text as this concentration inequality is the main theoretical message. All results establishing the Chebbyshev inequality (Thm 3.11, Lemma 3.12, Lemma 3.14 requirements should go to the appendix. Rather, the implications of the Theorem 3.13 should be elaborated on more. I appreciate that the authors note that these bounds are extremely loose.
Theory.
-Theorem 3.13 is a good first step to understanding the approximation quality (and indeed consistency) of the algorithm. It is, however, not useful in the sense that we do not care about the kernel approximation itself, but we care about the generalization error in the downstream algorithm. Kernel method theory has seen a substantial amount of work in that respect (e.g. for Nystrom, Fourier Feature regression/classification). This should be acknowledged, or even better: established. A simple approach would be perturbation analysis using the established kernel approximation error, better would be  directly controlling the error of the estimated decision boundary of the SVM.
Experiments.
-Runtime should be checked for various m
-Where does the approximation fail? An instructive synthetic example would help understanding when the algorithm is appropriate and when it is not.
-The authors mention that their algorithm allows for previously impossible settings of (k,m). In Table 3, however, there is only a single case where they demonstrate an improved performance as opposed to the exact algorithm (and the improvement is marginal). Either the authors need to exemplify the statement that their algorithm allows to solve previously unsolved problems (or allow for better accuracy), or they should remove it.
Minor.
-Figure 1 has no axis units
-Typo line 95 "theset"
-Typo line 204 "Generating these kernels days;"