An Active Learning Framework using Sparse-Graph Codes for Sparse Polynomials and Graph Sketching
Let f : {−1, 1} n → R be an n-variate polynomial consisting of 2 n monomials, in which only s 2 n coefficients are non-zero. The goal is to learn the polynomial by querying the values of f . We introduce an active learning framework that is associated with a low query cost and computational runtime. The significant savings are enabled by leveraging sampling strategies based on modern coding theory, specifically, the design and analysis of sparse-graph codes, such as Low-Density-Parity-Check (LDPC) codes, which represent the state-of-the-art of modern packet communications. More significantly, we show how this design perspective leads to exciting, and to the best of our knowledge, largely unexplored intellectual connections between learning and coding. The key is to relax the worst-case assumption with an ensemble-average setting, where the polynomial is assumed to be drawn uniformly at random from the ensemble of all polynomials (of a given size n and sparsity s). Our framework succeeds with high probability with respect to the polynomial ensemble with sparsity up to s = O(2 δn ) for any δ ∈ (0, 1), where f is exactly learned using O(ns) queries in time O(ns log s), even if the queries are perturbed by Gaussian noise. We further apply the proposed framework to graph sketching, which is the problem of inferring sparse graphs by querying graph cuts. By writing the cut function as a polynomial and exploiting the graph structure, we propose a sketching algorithm to learn the an arbitrary n-node unknown graph using only few cut queries, which scales almost linearly in the number of edges and sub-linearly in the graph size n. Experiments on real datasets show significant reductions in the runtime and query complexity compared with competitive schemes.
The paper studies the problem of actively learning a function f:{-1,1}^n -> R which is a sum of at most s parity functions or in other words its Fourier spectrum is sparse. The algorithm is allowed to query the value of the function at any desired point z in F2^n.       The main result is: For any function whose parity functions are randomly uniformly drawn from the set of all parities and whose coefficients are drawn from a distribution on a bounded set, the algorithm runs in time O(ns logs) with O(ns) queries and learns the function. This is also possible when the queries are noisy.   Another special case of the result is when the parities have at most d variables in them. In this case, the algorithm has O(sd logs logn) query complexity and O(ds log2s logn) run time. Further, this is applied to the problem of learning hypergraphs from cuts (graph sketching) and when the hyperedges are of constant size, the algorithm learns the graph in O*(r) time where O*() involves some poly log terms in n and r. This is strictly better than any algorithm known all of which needs poly in n running time. The results have been compared to another set of algorithms developed for the case of sparse function learning from random samples. Simulations on real data sets have been provided that demonstrate the gain from other algorithms. Main Ideas:Actually, in finding coefficients of parities and in also finding the parities, this work designs queries in such a way that a "sparse parity-check like equations" akin to decoding LDPC codes in coding theory is formed and peeling decoders are used to find the solution. In detail the work uses the following two ideas:Idea 1: Fourier expansion of f is a Walsh-Hadamard Transform with the coefficients of parities serving as the WHT coefficients. Now if one takes the s- point WHT (WHT is also its inverse) of a bunch of queries evaluated on a hyperplane H in F2n of dimension b = O(log s),then each of the coefficients of the s-point WHT is a very sparse linear combination of only a subset of coefficients of parities. The s-point WHT of the queries and the coefficients of the actual parities form a sparse bipartite graph where peeling decoding is employed. However, this can reveal only the coefficients of the parities but not the parities themselves. For that authors employ the following ideaIdea 2: Each of the s-point WHT of the queries is formed by linearly combining +1s and -1s and the sign depends only on the shift of the hyperplane H from origin and the parity.So multiple hyperplanes are used to create different +-1 linear combinations. Now these can be used at the time of peeling, to relate the actual parity and the sign of the s-point WHTs for different hyperplanes.So O(n) hyperplanes are necessary in general. But if the parities are of bounded degree d, then clearly clever design of the hyperplane shifts can result in another compresses sensing problem with bi-partite graphs and peeling decoding which will reduce the number of hyper planes requires to O(d log n). Strengths: The results for bounded degree parities and its subsequent application graph sketching are novel and they almost match the information theoretic lower bound of O*(r) query complexity for the graph sketching problem. The simulations are very interesting and its comparison with previous algorithm for the random examples case illustrates its utility.Weaknesses: 1) This is my most important comment/suggestion:In section 1.2 and in several places, the solution of this work is compared with ref16, ref21 and ref9. But those algorithms are for random samples with no query access. However, the Query access model is a lot more powerful. So I think the authors could try comparing it with Kushilevitz-Mansour algorithm (refer to Thm 3.37 in the Book:Analysis of Boolean functions by O'Donnell). Kushilevitz-Mansour algorithm (with some very simple modifications to the case when f:{-1,1}n-> R) can be made to run in time poly(n,s) and it also designs queries adaptively. Closely, looking at it, it seems like it has very similar queries and run time complexities although I could be wrong. However, the bounded degree case seems novel and the resulting graph sketching application seems novel. But even in this case, the algorithm given in page 85, of the same book, could be modified to start with buckets of size nd instead of size 2n as in conventional Kushilevitz-Mansour algorithm and I think that would give the O(d log n) scaling that accompanies the results in the paper.However, I am not very sure about the second suggestion. Authors could clarify this point.2) Minor comments:a) Choices for matrices Mc in page (4) is not mentioned in the main paper. It seems like it should not matter much. But I am not sure what role it plays in the peeling decoding analysis.b) Proposition 1 "...where each group generates P a query set.. "-> 'a' should be removed. c) g_{\hat{k}} in algorithm 1 is not defined in the main paper. d) Last but the second line before Definition 3: "that is is ..." -> only one 'is'. e) \mathbf{H} is defined using MATLAB notation of concatenating matrices row wise. Could clarify that notation somewhere. f) Section 3 Line 4 - "..cut xcan be..." - > needs to change.