REINFORCEMENT LEARNING THROUGH NEURAL ENCODING
Recent progress in the field of Reinforcement Learning (RL) has enabled to tackle bigger and more challenging tasks. However, the increasing complexity of the problems, as well as the use of more sophisticated models such as Deep Neural Networks (DNN), has impeded the ability to understand the behavior of trained policies. In this work, we present the Semi-Aggregated Markov Decision Process (SAMDP) model. The purpose of the SAMDP modeling is to analyze trained policies by identifying temporal and spatial abstractions. In contrast to other modeling approaches, SAMDP is built in a transformed state-space that encodes the dynamics of the problem. We show that working with the right state representation mitigates the problem of finding spatial and temporal abstractions. We describe the process of building the SAMDP model by observing trajectories of a trained policy and give examples for using it in a toy problem and complicated DQN agents. Finally, we show how using the SAMDP we can monitor the trained policy and make it more robust.
INTRODUCTION
In the early days of RL, understanding the behavior of trained policies could be done rather easily (Sutton, 1990). Researchers focused on simpler problems (Peng and Williams, 1993), and policies were built using lighter models than today (Tesauro, 1994). As a result, a meaningful analysis of policies was possible even by working with the original state representation and relating to primitive actions. However, in recent years research has made a huge step forward. Fancier models such as Deep Neural Networks (DNNs) have become a commodity (Mnih et al., 2015), and the RL community tackles bigger and more challenging problems (Silver et al., 2016). Artificial agents are even expected to be used in autonomous systems such as self-driving cars. The need to reason the behavior of trained agents, and understand the mechanisms that govern its choice of actions is pressing more than ever.
Analyzing a trained policy modeled by a DNN (either graphically using the state-action diagram, or by any other mean) is practically impossible. A typical problem consists of an immense number of states, and policies often rely on skills (Mankowitz, Mann, and Mannor, 2014), creating more than a single level of planning. The resulting Markov reward processes induced by such policies are too complicated to comprehend through observation. Simplifying the behavior requires finding a suitable representation of the state space; a long-standing problem in machine learning, where extensive research has been conducted over the years (Boutilier, Dean, and Hanks, 1999). There, the goal is to come up with a transformation of the state space φ : s → ŝ, that can facilitate learning.
∗These authors contributed equally
In the field of RL, where problems are sequential in nature, this problem is exacerbated since the representation of a state needs to account for the dynamics of the problem as well.
Finding a suitable state representation can be phrased as a learning problem itself (Ng, 2011; Lee et al., 2006). DNNs are very useful in this context since they automatically build a hierarchy of representations with an increasing level of abstraction along the layers. In this work, we show that the state representation that is learned automatically by DNNs is suitable for building abstractions in RL. To this end, we introduce the SAMDP model; a modeling approach that creates abstractions both in space and time. Contrary to other modeling approaches, SAMDP is built in a transformed state space, where the problem of creating spatial abstractions (i.e., state aggregation), and temporal abstractions (i.e., identifying skills) is facilitated using spatiotemporal clustering. We provide an example for building an SAMDP model for a basic gridworld problem where φ(s) is hand-crafted. However, the real strength of the model is demonstrated on challenging Atari2600 games solved using DQNs (Mnih et al., 2015). There, we set φ(s) to be the state representation automatically learned by the DNN (i.e. the last hidden layer). We continue by presenting methods for evaluating the fitness of the SAMDP model to the trained policy at hand. Finally, we describe a method for using the SAMDP as a monitor that alerts when the policy’s performance weakens, and provide initial results showing how the SAMDP model is useful for shared autonomy systems.
BACKGROUND
We briefly review the standard reinforcement learning framework of discrete-time finite Markov decision processes (MDPs). An MDP is defined by a five-tuple < S,A, P,R, γ >. At time t an agent observes a state st ∈ S, selects an action at ∈ A, and receives a reward rt. Following the agent’s action choice, it transitions to the next state st+1 ∈ S according to a Markovian probability matrix Pa ∈ P . The cumulative return at time t is given by Rt = ∑∞ t′=t γ
t′−trt, where γ ∈ [0, 1] is the discount factor. In this framework, the goal of an RL agent is to maximize the expected return by learning a policy π : S → ∆A; a mapping from states s ∈ S to a probability distribution over actions. The action-value function Qπ(s, a) = E[Rt|st = s, at = a, π] represents the expected return after observing state s, taking action a and then following policy π. The optimal action-value function obeys a fundamental recursion known as the optimal Bellman Equation: Q∗(st, at) = E [ rt + γmax
a′ Q∗(st+1, a
′) ] .
Skills, Options (Sutton, Precup, and Singh, 1999) are temporally extended control structures, denoted by σ. A skill is defined by a triplet: σ =< I, π, β >, where I defines the set of states where the skill can be initiated, π is the intra-skill policy, and β is the set of termination probabilities determining when a skill will stop executing. β is typically either a function of state s or time t. Any MDP with a fixed set of skills is a Semi-MDP (SMDP). Planning with skills can be performed by learning for each state the value of choosing each skill. More formally, an SMDP is defined by a five-tuple < S,Σ, P,R, γ >. S is the set of states, Σ is the set of skills, P is the SMDP transition
matrix, γ is the discount factor and the SMDP reward is defined by:
Rσs = E[rσs ] = E[rt+1 + γrt+2 + · · ·+ γk−1rt+k|st = s, σ]. (1)
The Skill Policy µ : S → ∆Σ is a mapping from states to a distribution over skills. The action-value function Qµ(s, σ) = E[ ∑∞ t=0 γ
tRt|(s, σ), µ] represents the value of choosing skill σ ∈ Σ at state s ∈ S, and thereafter selecting skills according to policy µ. The optimal skill value function is given by: Q∗Σ(s, σ) = E[Rσs + γkmax
σ′∈Σ Q∗Σ(s ′, σ′)] (Stolle and Precup, 2002).
THE SEMI AGGREGATED MDP
Reinforcement Learning problems are typically modeled using the MDP formulation. Given an MDP, a variety of algorithms have been proposed to find an optimal policy. However, when one wishes to analyze a trained policy, MDP may not be the best modeling choice due to the size of the state space and the length of the planning horizon. In this section, we present the SMDP and Aggregated MDP (AMDP) models which can simplify the analysis by using temporal and spatial abstractions respectively. We also introduce the new Semi-Aggregated MDP (SAMDP) model, that combines SMDP and AMDP models in a novel way which leverages the abstractions made in each modeling approach. SMDP (Sutton, Precup, and Singh, 1999), can simplify the analysis of a trained policy by using temporal abstractions. The model extends the MDP action space A to allow the agent to plan with temporally extended actions Σ (i.e., skills). Analyzing policies using the SMDP model shortens the planning horizon and simplifies the analysis. However, there are two problems with this approach. First, one still faces the high complexity of the state space, and second, the SMDP model requires to identify skills.
Skill identification is an ill-posed problem that can be addressed in many ways, and for which extensive research has been done over the years. The popular approaches are to identify bottlenecks in the state space (McGovern and Barto, 2001),or to search for common behavior trajectories, or common state region policies (McGovern, 2002). A different approach can be to build a graphical model of the agent’s interaction with the environment and to use betweenness centrality measures to identify subtasks (Şimşek and Barreto, 2009). No matter what the method is, identifying skills solely by observing an agent play is a challenging task.
Alternative approach to SMDP modeling is to analyze a policy using spatial abstractions in the state space. If there is a reason to believe that groups of states share common attributes such as similar policy or value function, it is possible to use State Aggregation (Moore, 1991). State Aggregation is a well-studied problem that typically involves identifying clusters as the new states of an Aggregated MDP, where the set of clusters C replaces the MDP states S. Applying RL on aggregated states is potentially advantageous because the dimensions of the transition probability matrix P , the reward signal R and the policy π are decreased (Singh, Jaakkola, and Jordan, 1995). However, the AMDP modeling approach has two drawbacks. First, the action space A is not modified, and therefore the planning horizon remains intractable, and second, AMDPs are not necessarily Markovian (Bai, Srivastava, and Russell, 2016). In this paper, we propose a model that combines the advantages of the SMDP and AMDP approaches and denote it by SAMDP. Under SAMDP modeling, aggregation defines both the states and the set of skills, allowing analysis with spatiotemporal abstractions (the state-space dimensions and the planning horizon are reduced). However, SAMDPs are still not necessarily Markovian. We summarize the different modeling approaches in Figure 1. The rest of this section is devoted to explaining the five stages of building an SAMDP model: (0) Feature selection, (1) State Aggregation, (2) Skill identification, (3) Inference, and (4) Model Selection. (0) Feature selection. We define the mapping from MDP states to features, by a mapping function φ : s → s′ ⊂ Rm. The features may be raw (e.g., spatial coordinates, frame pixels) or higher level abstractions (e.g., the last hidden layer of an NN). The feature representation has a significant effect on the quality of the resulting SAMDP model and vice versa; a good model can point out a good feature representation. (1) Aggregation via Spatio-temporal clustering. The goal of Aggregation is to find a mapping (clustering) from the MDP feature space S′ ⊂ Rm to the AMDP state space C. Clustering algorithms typically assume that data is drawn from an i.i.d distribution. However, in our problem data
is generated from an MDP which violates this assumption. We alleviate this problem using two different approaches. First, we decouple the clustering step from the SAMDP model, by creating an ensemble of clustering candidates and building an SAMDP model for each (following stages 2 and 3). In stage 4, we will explain how to run a non-analytic outer optimization loop to choose between these candidates based on spatiotemporal evaluation criteria. Second, we introduce a novel extension of the celebrated K-means algorithm (MacQueen and others, 1967), which enforces temporal coherency along trajectories. In the vanilla K-means algorithm, a point xt is assigned to cluster ci with mean µi if µi is the closest cluster center to xt (for further details please see the supplementary material). We modified this step as follows:
c(xt) = { ci : ∥∥Xt − µi∥∥2F ≤ ∥∥Xt − µj∥∥2F ,∀j ∈ [1,K]}, where F stands for the Frobenius norm, K is the number of clusters, t is the time index of xt, and Xt is a set of 2w+ 1 centered at xt from the same trajectory: { xj ∈ Xt ⇐⇒ j ∈ [t−w, t+w] } . The dimensions of µ correspond to a single point, but is expanded to the dimensions of Xt. In this way, we enforce temporal coherency since a point xt is assigned to a cluster ci if its neighbors in time along the trajectory are also close to µi. We have also experimented with other clustering methods such as spectral clustering, hierarchical agglomerative clustering and entropy minimization (please refer to the supplementary material for more details). (2) Skill identification. We define an SAMDP skill σi,j ∈ Σ uniquely by a single initiation state ci ∈ C and a single termination state cj ∈ C : σij =< ci, πi,j , cj > . More formally, at time t the agent enters an AMDP state ci at an MDP state st ∈ ci. It chooses a skill according to its SAMDP policy and follows the skill policy πi,j for k time steps until it reaches a state st+k ∈ cj , s.t i 6= j. We do not define the skill length k apriori nor the skill policy but infer the skill length from the data. As for the skill policies, our model does not define them explicitly, but we will observe later that our model successfully identifies skills that are localized in time and space.
(3) Inference. Given the SAMDP states and skills, we infer the skill length, the SAMDP reward and the SAMDP probability transition matrix from observations. The skill length, is inferred for a skill σi,j by averaging the number of MDP states visited since entering SAMDP state ci until leaving for SAMDP state cj . The skill reward is inferred similarly using Equation 1. The inference of the SAMDP transition matrices is a bit more puzzling, since the probability of seeing the next SAMDP state depends both on the MDP dynamics and the agent policy in the MDP state space. We now turn to discuss how to infer these matrices by observing transitions in the MDP state space. Our goal is to infer two quantities: (a) The SAMDP transition probability matrices PΣ : P σ∈Σ i,j = Pr(cj |ci, σ), measures the probability of moving from state ci to cj given that skill σ is chosen. These matrices are defined uniquely by our definition of skills as deterministic probability matrices. (b) The probability of moving from state ci to cj given that skill σ is chosen according to the agent SAMDP policy: Pπi,j = Pr(cj |ci, σ = π(ci)). This quantity involves both the SAMDP transition probability matrices and the agent policy. However, since SAMDP transition probability matrices are deterministic, this is equivalent to the agent policy in the SAMDP state space. Therefore by inferring transitions between SAMDP states, we directly infer the agent’s SAMDP policy. Given an MDP with a deterministic environment and an agent with a nearly deterministic MDP policy (e.g., a deterministic policy that uses an -greedy exploration ( 1)), it is intuitive to assume that we would observe a nearly deterministic SAMDP policy. However, there are two mechanisms that cause stochasticity in the SAMDP policy: (1) Stochasticity that is accumulated along skill trajectories. (2) Approximation errors in the aggregation process. A given SAMDP state may contain more than one ”real” state and therefore more than one skill. Performing inference in this setup, we might observe a stochastic policy that chooses randomly between skills. Therefore, it is very likely to infer a stochastic SAMDP transition matrix, even though the SAMDP transition probability matrices and the MDP environment are deterministic, and the MDP policy is nearly deterministic. (4) Model selection. So far we have explained how to build an SAMDP from observations. In this stage, we’ll explain how to choose between different SAMDP model candidates. There are two advantages of choosing between multiple SAMDPs. First, there are different hyperparameters to tune: two examples are the number of SAMDP states (K) and the window size (w) for the clustering algorithm. Second, there is randomness in the aggregation step. Hence, clustering multiple times and picking the best result will potentially yield better models.
We developed, therefore, evaluation criteria that allow us to select the best model, motivated by Hallak, Di-Castro, and Mannor (2013). We follow the Occams Razor principle and aim to find the simplest model which best explains the data. (i) Value Mean Square Error(VMSE), measures the consistency of the model with the observations. The estimator is given by
VMSE = ‖v−vSAMDP ‖/‖v‖, (2)
where v stands for the SAMDP value function of the given policy, and vSAMDP is given by: VSAMDP = (I + γ
kP )−1r, where P is measured under the SAMDP policy. (ii) Inertia, the Kmeans algorithm objective function, is given by : I = ∑n i=0 minµj∈C(||xj − µi||2). Inertia measures the variance inside clusters and encourages spatial coherency. Motivated by Ncut and spectral clustering (Von Luxburg, 2007), we define (iii) The Intensity Factor as the fraction of out/in cluster transitions. However, we define edges between states that are connected along the trajectory (a transition between them was observed) and give them equal weights (instead of defining the edges by euclidean distances as in spectral clustering). Minimizing the intensity factor encourages longer duration skills. (iv)had been Entropy, is defined on the SAMDP probability transition matrix as follows: e = − ∑ i{|Ci| · ∑ j Pi,j logPi,j}. Low entropy encourages clusters to have less skills, i.e., clusters that are localized both in time and space.
SAMDP FOR GRIDWORLD
We first illustrate the advantages of SAMDP in a basic gridworld problem (Figure 2). In this task, an agent is placed at the origin (marked in X), where the goal is to reach the green ball and return. The state s ∈ R3 is given by: s = {x, y, b}, where (x, y) are the coordinates and b ∈ {0, 1} indicates whether the agent has reached the ball or not. The policy is trained to find skills following the algorithm of Mankowitz, Mann, and Mannor (2014). We are given trajectories of the trained agent, and wish to analyze its behavior by building the state-action graph for all four modeling approaches. For clarity, we plot the graphs on the maze using the coordinates of the state. The MDP graph (Figure 2(a)), consists of a vast number of states. It is also difficult to understand what skills the agent is using. In the SMDP graph (Figure 2(b)), the number of states remain high, however
coloring the edges by the skills, helps to understand the agent’s behavior. Unfortunately, producing this graph is seldom possible because we rarely receive information about the skills. On the other hand, abstracting the state space can be done more easily using state aggregation. However, in the AMDP graph (Figure 2(c)), the clusters are not aligned with the played skills because the routes leading to and from the ball overlap. For building the SAMDP model (Figure 2(d)), we transform the state space in a way that disentangles the routes:
φ(x, y) = { (x, y), if b is 0 (2L− x, y), if b is 1 ,
where L is the maze width. The transformation φ flip and translate the states where b = 1. Now that the routes to and from the ball are disentangled, the clusters are perfectly aligned with the skills. Understanding the behavior of the agent is now possible by examining inter-cluster and intra-cluster transitions.
SAMDPS FOR DQNS
Feature extraction: We evaluate a pre-trained DQN agent for multiple trajectories with an -greedy policy on three Atari2600 games, Pacman (a game where DQN performs very well), Seaquest (for the opposite reason) and Breakout (for its popularity). We let the trained agent play 120k game states, and record the neural activations of the last hidden layer as well as the Q values. We also keep the time index of each state to be able to find temporal neighbors. Features from other layers can also be used. However, we rely on the results from Zahavy, Zrihem, and Mannor (2016) that showed that the features learned in the last hidden layer capture a spatiotemporal hierarchy and therefore make a good candidate for state aggregation. We then apply t-SNE on the neural activations data, a non-linear dimensionality reduction method that is particularly good at creating a single map that reveals structure at many different scales. We use the two coordinates of the t-SNE map and the value estimation as the MDP state features. Each coordinate is normalized to have zero mean and
unit variance. We have experimented with other configurations such as using the activations without t-SNE as well as different normalization. However, we found that this configuration results in better SAMDP models. We also use two approximations in the inference stage which we found to work well: 1) overlooking transitions with a small skill length (shorter than 2) and 2) truncating transitions with probability less than 0.1. We only present results for the Breakout game and refer the reader to the supplementary material for results on Pacman and Seaquest. Model Selection: We perform a grid search on two parameters: i) number of clusters K ∈ [15, 25]. ii) window size w ∈ [1, 7]. We found that models larger (smaller) than that are too cumbersome (simplistic) to analyze. We select the best model in the following way: we first sort all models by the four evaluation criteria (SAMDP Section, stage 4) from best to worst. Then, we iteratively intersect the p-prefix of all sets (i.e., the first p elements of each set) starting with 1-prefix. We stop when the intersection is nonempty and choose the configuration at the intersection. The resulted SAMDP model for Breakout can be seen in Figure 3. We also measure the p-value of the chosen model. For the null hypothesis, we take the SAMDP model constructed with random clusters. We tested 10000 random SAMDP models, none of which scored better than the chosen model (for all the evaluation criteria). Qualitative Evaluation: Examining the resulting SAMDP (Figure 3) it is interesting to note the sparsity of transitions, which implies low entropy. Inspecting the mean image of each cluster reveals insights about the nature of the skills hiding within and uncovers the policy hierarchy as described in Zahavy, Zrihem, and Mannor (2016). The agent begins to play in low value (blue) clusters (e.g., 1,5,8,9,13,16,18,19). These clusters are well connected between them and are disconnected from other clusters. Once the agent transitions to the ”tunnel-digging” option in clusters 4,12,14, it stays in there until it finishes to curve the tunnel, then it transitions to cluster 11. From cluster 11 the agent progresses through the ”left banana” and hops between clusters 2,21,5,10,0,7 and 3 in that order. Model Evaluation: We first measure the VMSE criterion, as defined in Equation 2 (Figure 4, top). We infer v by averaging the DQN value estimates in each cluster: vDQN (cj) =
1 |Cj | ∑ i:si∈cj v
DQN (si), and evaluate VSAMDP as defined above. Since the Atari environment is deterministic, the vSAMDP estimate is accurate with respect to the DQN policy. Therefore, the VMSE criterion measures how well the SAMDP model approximates the true MDP. In practice, we observe that the DQN and SAMDP values are very similar; indicating that the SAMDP model fits the data well. Second, we evaluate the greedy policy with respect to the SAMDP value function by: πgreedy(ci) = argmax
j {Rσi,j + γ kσi,j vSAMDP (cj)}. We then measure the correlation between the greedy policy decisions and the trajectory reward. For a given trajectory j we measure P ji : the empirical distribution of choosing the greedy policy at state ci and the cumulative reward Rj . Finally, we present the correlation between these two measures in each state: corri = corr(P j i , R j) in (Figure 4, center). A
positive correlation indicates that following the greedy policy leads to high reward. Indeed for most of the states, we observe positive correlation, supporting the consistency of the model. The third evaluation is close in spirit to the second one. We partition the data to a train and test sets. We evaluate the greedy policy based on the train set and create two transition matrices T+, T− using the k top and bottom rewarded trajectories respectively from the test set. We measure the correlation of the greedy policy TG with each of the transition matrices for different values of k (Figure 4 bottom). As clearly seen, the correlation of the greedy policy and the top trajectories is higher than the correlation with the bottom trajectories. Eject Button: The motivation for this experiment stems from the idea of shared autonomy Pitzer et al. (2011). There are domains where errors are dreadful, and performance must be as high as possible. The idea of shared autonomy, is to allow an operator to intervene in the decision loop at critical times. For example, in 20% of commercial flights, the auto-pilot returns the control to the human pilots. In the following experiment, we show how the SAMDP model can help to identify where the agent’s behavior deteriorates. Setup. (a) Evaluate a DQN agent, create a trajectory data set, and evaluate the features for each state (stage 0). (b) Divide the data into two groups: train (100 trajectories) and test (60). then build an SAMDP model (stages 1-4) on the train data. (c) Split the train data to k top and bottom rewarded trajectories T+, T− and re-infer the model parameters separately for each (stage 3). (d) Project the test data on the SAMDP model (mapping each state to the nearest SAMDP state). (e) Eject when the transitions of the agent are more likely under the T− matrix rather then under T+ (inspired by the idea of option interruption Sutton, Precup, and Singh (1999)). (f) We average the trajectory reward on (i) the entire test set, and (ii) the un-ejected trajectories sub set. We measure 36% ± 7.7%, 20% ± 8.0%, and 4.7% ± %1.2 performance gain for Breakout Seaquest and Pacman, respectively. The eject experiment indicates that the SAMDP model can be used to make a given DQN policy robust by identifying when the agent is not going to perform well and return control to a human operator or some other AI agent. Other eject mechanisms are also possible. For example, ejecting by looking at MDP values. However, the Q value is not monotonically decreasing along the trajectory as expected (See Figure 3). The solution we propose is to eject by monitoring transitions and not state values, which makes the MDP impractical in this case because it’s state-action diagram is too large to construct, and too expensive to process.
DISCUSSION
SAMDP modeling offers a way to present a trained policy in a concise way by creating abstractions that relate to the spatiotemporal structure of the problem. We showed that by using the right representation, time-aware state aggregation could be used to identify skills. It implies that the crucial step in building an SAMDP is the state aggregation phase. The aggregation depends on the state features and the clustering algorithm at hand.
In this work, we presented a basic K-means variant that relies on temporal information. However, other clustering approaches are possible. We also experimented with agglomerative methods but found them to be significantly slower without providing any benefit. We believe that clustering methods that better relate to the topology, such as spectral clustering, would produce the best results. Regarding the state features; in the DQN example, we used the 2D t-SNE map. This map, however, is built under the i.i.d assumption that overlooks the temporal dimension of the problem. An interesting line of future work will be to modify the t-SNE algorithm to take into account temporal distances as well as spatial ones. A tSNE algorithm of this kind may produce 2D maps with even lower entropy which will decrease the aggregation artifacts that affect the quality of the SAMDP model.
In this work we analyzed discrete-action policies, however SAMDP can also be applied for continuous-action policies that maintain a value function (since our algorithm depends on it for construction and evaluation), as in the case of actor-critic methods. Another issue we wish to investigate is the question of consistency in re-building an SAMDP. We would like the SAMDP to be unique for a given problem. However, there are several aspects of randomness that may cause divergence. For instance, when using a DQN, randomness exists in the creation of the t-SNE map, and in the clustering phase. From our experience, though, different models built for the same problem are reasonably consistent. In future work, we wish to address the same problem by laying out an optimization problem that will directly account for all of the performance criteria introduced here. It would be interesting to see what clustering method will be drawn out of this process and to compare the principled solution with our current approach.

The paper presents a method for visualization and analysis of policies from observed trajectories that the policies produce. The method infers higher level skills and clusters states. The result is a simplified, discrete higher-order state and action transition matrix. This model can be used for analysis, modeling, and interpretation. To construct semi-aggregated MDP the authors propose combining ideas for creating semi-MPDs and agregrated-MDPs. The method consists of choosing features, state clustering, skill inference, reward and skill length inference, and model selection. The method was demonstrated on a small grid-world problem, and DQN-trained agent for playing Atari games.
The authors correctly identify that tools and means for interpretibility of RL methods are important for analysis, and deployment of such methods for real-world applications. This is particularly true in robotics and high-consequence systems.
The end-result of the presented method is a high-level transition matrix. There is a big body of literature looking into hierarchical RL methods where lower level skills are combined with higher level policies. The presented method has the similar result, but the advantage of the presented method is that it comes up with a structure and analyzes already trained agent, which is very interesting. The paper would benefit from emphasizing this difference, and contrasting with the broader body of literature.
To build the model, the authors propose combining the ideas from two existing ideas, semi-MPDs and agregrated-MDPs with using modified k-means for state clustering. It appears that the novelty of the presented method is limited. The paper would have been stronger if the authors explicitly stated the contributions over combining existing methods, and better highlighted the practical utility of the method.
The evaluation section would be made stronger with more analytical results and precise evaluation, showing full strength of the method.
The paper is difficult to read. To improve readability:
- The Semi-Aggregated MDP section should include more precise description of the methods. The narrative that builds intuition is welcome. In addition to the existing narrative, algorithms and formulas where applicable should be included as well.
- The paper should be self-contained. For example, more background on Occams Razor principle should be included.
- Reduce the number of acronyms, in particular similarly sounding acronyms. Define acronyms before using.
- Be more clear on the contributions, contrast with relevant literature, and the specific benefits of the presented method.
- Fix typos, formatting mistakes etc., as they can be distracting for reading.
The approach of reverse engineering the hierarchy, and learning high-level transition matrix is very interesting and promising. Perhaps the method can be used to outperform single network approach by using the model as an input to more specialized hierarchical trainers and learn complex behaviors more optimally then possible with one large network approach. Unfortunately, the paper falls short in the novelty, precision, and clarity.


Parallel Feature Selection inspired by Group Testing
This paper presents a parallel feature selection method for classification that scales up to very high dimensions and large data sizes. Our original method is inspired by group testing theory, under which the feature selection procedure consists of a collection of randomized tests to be performed in parallel. Each test corresponds to a subset of features, for which a scoring function may be applied to measure the relevance of the features in a classification task. We develop a general theory providing sufficient conditions under which true features are guaranteed to be correctly identified. Superior performance of our method is demonstrated on a challenging relation extraction task from a very large data set that have both redundant features and sample size in the order of millions. We present comprehensive comparisons with state-of-the-art feature selection methods on a range of data sets, for which our method exhibits competitive performance in terms of running time and accuracy. Moreover, it also yields substantial speedup when used as a pre-processing step for most other existing methods.

This paper proposes a novel method for feature selection based on independent group testing, which is desirable for parallel execution for feature selection. The method consists of three steps: (1) test design, (2) score function, and (3) feature identification. In test design, groups of features for being tested are determined, score function assigns a score to each group, and feature identification step finally selects a set of features based on the scores. This paper presented a theory including the number of tests needed to maintain feature selection accuracy, and the experimental results which support that the proposed approach outperforms the competitors in terms of accuracy, scalability and robustness. Major comments:1. In line 132, groups were randomly generated. Are there advantages or disadvantages of making small or large groups?In test design, it would be useful if authors include more discussion on how groups of features should be made both in theory and practice.2. In experiments, as competing methods, popular sparse regression based feature selection methods should be included (e.g. Lasso or L1-logistic regression).3. In Figure 2, why does the proposed method improve the accuracy of FGM but not MIM when used as preprocessing step?4. It would be good to add more synthetic experiments to verify that theory is in agreement with the experiments.5. Conclusion or discussion section is missing. Minor comments:1. In line 82, there a typo. schem -> scheme 2. Section 2 and 3 can be structured for better readability. 3. In Figure 1, x-axis label is missing. 4. In Figure 2, two dotted lines overlap, and it is hard to distinguish between the two lines.


Sparse Polynomial Learning and Graph Sketching
Let f : {−1, 1} n → R be a polynomial with at most s non-zero real coefficients. We give an algorithm for exactly reconstructing f given random examples from the uniform distribution on {−1, 1} n that runs in time polynomial in n and 2 s and succeeds if the function satisfies the unique sign property: there is one output value which corresponds to a unique set of values of the participating parities. This sufficient condition is satisfied when every coefficient of f is perturbed by a small random noise, or satisfied with high probability when s parity functions are chosen randomly or when all the coefficients are positive. Learning sparse polynomials over the Boolean domain in time polynomial in n and 2 s is considered notoriously hard in the worst-case. Our result shows that the problem is tractable for almost all sparse polynomials.
Then, we show an application of this result to hypergraph sketching which is the problem of learning a sparse (both in the number of hyperedges and the size of the hyperedges) hypergraph from uniformly drawn random cuts. We also provide experimental results on a real world dataset.

The paper describes an algorithm for learning sparse polynomials over the boolean domain. It differs from works like [12,13] in that even though it has the same sample-complexity, it has much better run-time. It uses standard l1 minimization techniques but is able to perform a "feature-selection" step to reduce the combinatorial number of subsets from 2^n to 2^s, where s is the sparsity of the polynomial.Of all the 5 other paper I reviewed for NIPS, this was the best one. However, it was also the slowest for me to read and I am the least certain about it, mainly because it was the paper farthest from my own specialty. There were quite a few arguments that I couldn't check in detail and instead rely on trusting the authors.The writing was clear, and the intuition was normally laid out well. I was at first disappointed that Lemma 2 and Theorem 4 were relegated to the appendix, but the appendix was short and I think this was OK.Overall, assuming I didn't misinterpret anything, I'm enthusiastic about publishing this paper. There was more in this paper than in most of the other papers I reviewed combined. I am a bit confused why this paper is in NIPS and not STOC, though.My minor criticisms are below:- In the abstract, the description of the unique sign property was confusing, even after reading the definition in the paper. The definitions in the paper seemed good, but not the one in the abstract.- I didn't notice the sample complexity mentioned in the first part of the paper. It is O(n), right? Please state this earlier, since it is of great interest (if it was 2^s, then a fast time complexity is sort of irrelevant)- Theorem 2: parity functions are not yet defined, so this is not yet clear to a non-expert like myself. Similarly, section 2 was a good first-step, but I still needed to supplement this with some basic notes about Fourier expansion of boolean functions.- Section 1.1. The difference in the results with [12,13] and [15] was made quite clear, but the difference in techniques was not. I am very curious how much of the technique was novel. Please state the key technical innovation (or was the main innovation just the discovery of the algorithm/idea, and then the analysis was standard?).- Section 3 and later on, the notation for an ordered pair of samples and labels uses angular brackets, which are usually reserved for inner products. Why not just use parenthesis?- Theorem 3 and following paragraph refer to feasible "solutions" of the problem, which is confusing since I interpret "solution" and "optimum" identically, and with this interpretation it makes no sense. I would say a "feasible point", not a "feasible solution".- The technique reminds me of "Safe feature elimination" by El Ghaoui, who proves a similar reduction technique for l1 minimization (in the standard compressed sensing context), but of course quite different since his variables are not linked the same way yours are.- The step of finding feasible solutions p in equation 4 in Algo 1 is not explained in the main text, so this is a bit of a mystery. It would be nice to give more intuition here. (Here, it is OK to call p a "solution" since you are just solving an equation, not solving an equation AND minimizing).- Section 5 was either written by a different co-author or not proof-read to the same degree, because the writing quality deteriorates and there are many English mistakes. Bottom of page 5, "previously" is used twice in a sentence.  There are many missing "the"'s in the sentences.- After equation 6, what is "d"?- Theorem 5 and this specialization is quite nice. However, it felt secondary so I did not check the corresponding section in the appendix (except to note that the proof of Thm 5 is hard to follow and condensed into 1 giant paragraph, very unlike the proof of Thm 4 and the lemma).- 5.2, the dataset was described, but it was never said what exactly it is (for people not familiar with Yahoo! Messenger). Is this an instant messaging, or something else? It wasn't clear to me the connection between instant messaging and zip codes, and I think you could explain this in 1 or 2 sentences to clear it all up.- In 5.2.1, you mention identifying the relevant variables correctly, but how do you know this? Was there a ground-truth solution? Did I miss something here?- The references should capitalize "Fourier" in the titles of papers.


A Filtering Approach to Stochastic Variational Inference
Stochastic variational inference (SVI) uses stochastic optimization to scale up Bayesian computation to massive data. We present an alternative perspective on SVI as approximate parallel coordinate ascent. SVI trades-off bias and variance to step close to the unknown true coordinate optimum given by batch variational Bayes (VB). We define a model to automate this process. The model infers the location of the next VB optimum from a sequence of noisy realizations. As a consequence of this construction, we update the variational parameters using Bayes rule, rather than a hand-crafted optimization schedule. When our model is a Kalman filter this procedure can recover the original SVI algorithm and SVI with adaptive steps. We may also encode additional assumptions in the model, such as heavytailed noise. By doing so, our algorithm outperforms the original SVI schedule and a state-of-the-art adaptive SVI algorithm in two diverse domains.

Stochastic variational inference (SVI) requires careful selection of a step size. This paper proposes a Kalman filter to set the step size automatically. The authors show that standard Gaussian KF does not satisfy the Robbins Munro criteria (and performs badly). They propose to apply a KF based on T-distributions, and show that this gives better results than standard SVI.The problem addressed in this paper is relevant. The solutions proposed are also quite interesting and seem useful. I have few theoretical and computational concerns which I have discussed below. The paper is clearly written. The work is original and significant. The numerical experiments shown are satisfactory.Below are some of my comments to help improve the quality of this paper.Eq. 7 and 8 involve inverting a matrix at every iteration of the algorithm. Since Q and R change every time, we cannot precompute the step sizes. Does this increase the computation significantly? I would expect that there is some increase due to inversion of the matrices who size depends on size of lambda. Will you please discuss the extent of computation increase you notice in your experiments?Doing filtering and parameter estimation at the same time for Kalman filter should be problematic, at least initially. I don't see that in the Fig. 4. Why is that? Do you use some method to compensate for this? In any case, it will be good to include a discussion about the consequences of this approach. When is it going to work and when is it going to be troublesome? Does the size of lambda play a role in this too? etc.You mention that for fixed Q and R Robbins-Munro criteria do not hold. When they are varying, does it holds for sure? can you prove it? Is it the same for T-distribution KF? Is this the reason why in Fig. 4 Gaussian KF doesn't reach the true maximum? There are small typos here and there. There is a mistake in Eq. 5. The first equation works for t, while the second equation works with t+1.


Optimizing F-Measures by Cost-Sensitive Classification
We present a theoretical analysis of F -measures for binary, multiclass and multilabel classification. These performance measures are non-linear, but in many scenarios they are pseudo-linear functions of the per-class false negative/false positive rate. Based on this observation, we present a general reduction of Fmeasure maximization to cost-sensitive classification with unknown costs. We then propose an algorithm with provable guarantees to obtain an approximately optimal classifier for the F -measure by solving a series of cost-sensitive classification problems. The strength of our analysis is to be valid on any dataset and any class of classifiers, extending the existing theoretical results on F -measures, which are asymptotic in nature. We present numerical experiments to illustrate the relative importance of cost asymmetry and thresholding when learning linear classifiers on various F -measure optimization tasks.

The paper concerns a problem of training classifiers that maximize the F-measure. The authors show a reduction of the problem to cost-sensitive classification with unknown costs. They prove that such an approach is consistent and that in practical situations a good approximation can be obtained. The resulting algorithm follows an idea of training several cost-sensitive classifiers with different costs and choosing the one with the best F-measure. This is a quite different approach in comparison with the common solution in which a threshold maximizing the F-measure is tuned on a validation set. I like this paper, however, I have to admit that the structure of the paper could be improved. For example, I would rather prefer to present first the algorithm and its theoretical analysis for binary classification. Then, the authors could mention the extensions for other settings. I also tried to follow the proofs, but it was not always easy to do that. The authors should consider to move them to an appendix and present in a clearer way. The authors should also discuss wider the difference between the EUM (expected utility maximization) and DTA (decision-theoretic approach) frameworks, as these two are two different formulations of the problem. For example, the solution for DTA  does not necessarily converge to the solution of EUM for the micro F-measure.The introduced algorithm is very simple. Once we know that the optimal strategy is to find a threshold on conditional probabilities, we can either try to find the threshold on a validation set or use reduction to cost-sensitive classification with different costs (as in reduction of probability estimation to cost-sensitive binary classification). I would rather expect a more sophisticated strategy for finding the cost-sensitive problem with the best F-measure. Minor comments:- The sentence: "the optimal classifier (...) for the pseudo-linear F-measures is also an optimal classier for a cost-sensitive classification problem with label dependent costs" should be, in my opinion, stated in the other direction, i.e., "the optimal classifier for a cost-sensitive classification problem with label dependent costs is also optimal for the F-measure". The proof is in the right direction.- The authors should also present running times of the algorithms.After rebuttal:I thank the authors for their responses. I am looking forward for the revised version of the paper.


Extreme bandits
In many areas of medicine, security, and life sciences, we want to allocate limited resources to different sources in order to detect extreme values. In this paper, we study an efficient way to allocate these resources sequentially under limited feedback. While sequential design of experiments is well studied in bandit theory, the most commonly optimized property is the regret with respect to the maximum mean reward. However, in other problems such as network intrusion detection, we are interested in detecting the most extreme value output by the sources. Therefore, in our work we study extreme regret which measures the efficiency of an algorithm compared to the oracle policy selecting the source with the heaviest tail. We propose the EXTREMEHUNTER algorithm, provide its analysis, and evaluate it empirically on synthetic and real-world experiments.

This paper formulates an “extreme bandit” problem. An algorithm sequentially samples from among K distributions with the goal of observing a very extreme (i.e. large) realization. They propose a UCB-type algorithm for this problem, and establish a bound on its “extreme regret.”  This bound grows more slowly with the time horizon than the expected value of the most extreme observations of the optimal action, which implies that the algorithm concentrates on the best action asymptotically. This paper generalizes and strengthens prior results provided by [9,21]. The full parametric assumptions made in those works are replaced by more general assumptions about the tail behavior of reward distributions. In addition, this paper provides finite time rather than asymptotic guarantees. These are valuable extensions, but they feel like incremental progress. I think the paper is slightly below the acceptance threshold.I don’t find the motivation given in the introduction to be completely satisfying (see below), but in spite of this, it’s possible that there are many very important applications of these techniques with which I am not familiar. (Obviously anomaly detection is very important.)  I am somewhat uncertain of my review for this reason. Mathematical statements in the paper should be more carefully organized and presented in a clearer way. Section 5 tries to simultaneously define the algorithm and introduce a variety of constants that will be used in the theoretical analysis. I recommend defining the algorithm in a clear, self-contained way. You can move some analysis to the appendix if you need more space. Assumptions should be given in the problem formulation, or clearly set aside so the reader can keep track of them. Instead, a key technical assumption is added in the text in lines 213-215. It’s hard to track down all the constant terms, some of which are given only by looking at the paper [6] (right?). Further comments- By studying extreme regret, the paper does not precisely focus on identifying the distribution with the heaviest tails so much as on observing large realized values. Obviously these are related objectives, but they are not the same. Why choose to focus on extreme regret?- What are some natural examples where (1) the goal is to find extreme observations, (2) bandit feedback is a clear obstacle, (3) adaptive sampling is feasible, and (4) observations are costly enough that it’s worth designing very smart sampling schemes. - In the experiments, which estimator for h_{k,t} did you use?- The experimental results are averaged over only 100 trials, which seems particularly low given the focus on rare events. You should either increase the number of trials, or include some measure of the variability in the estimates. - Much of the motivation is that past work makes parametric assumptions that are too restrictive to be practical. It would be nice if the experiments then demonstrated the practical benefit of instead considering a semi-parametric approach.


Sensory Integration and Density Estimation
The integration of partially redundant information from multiple sensors is a standard computational problem for agents interacting with the world. In man and other primates, integration has been shown psychophysically to be nearly optimal in the sense of error minimization. An influential generalization of this notion of optimality is that populations of multisensory neurons should retain all the information from their unisensory afferents about the underlying, common stimulus [1]. More recently, it was shown empirically that a neural network trained to perform latent-variable density estimation, with the activities of the unisensory neurons as observed data, satisfies the information-preservation criterion, even though the model architecture was not designed to match the true generative process for the data [2]. We prove here an analytical connection between these seemingly different tasks, density estimation and sensory integration; that the former implies the latter for the model used in [2]; but that this does not appear to be true for all models.

This work investigates a generative model with latent variables. This model is assumed to perfectly generate some observed distribution, which was originally generated from some other "true" model with latent variables. Specifically, this work finds sufficient conditions so that any reasonable inference procedure done to infer the latent variables in the generative model from the observed variables would retain the information contained in these observed variables about the "true" latent variables.Quality: The core derivations (section 3.2+appendix) appear to be correct.Clarity: The organization of this manuscript seems rather fragmented and confusing, for a purely theoretical paper. It would benefit from clearly and explicitly stating in the beginning: (1) the general probabilistic model (2) all the assumptions, and (3) the main result. Also, some parts could be significantly shortened or removed (e.g. the first few paragraphs of section 2.1).Originality: As far as I am aware, these results are novel.Significance: It is not very clear to me how strong are these results. Most importantly, The assumptions of perfect observed density reconstruction, and of an identical noise model seem rather strong and limiting.Minor comments:1)Something seems to be missing from Eq. 3 ( \in what set?).2)Line 240 EHF->EFH%%% Edited after author feedback %%%Following the author response, I now agree that the "identical noise model" assumption is not as weak as I thought. This is just equivalent to the assumption that X "the state of the world" is deterministically converted into the direct input to the sensory neurons. Though this is probably not true, I agree that this can be considered as a reasonable first approximation.The relaxation of the other assumption ("perfect observed density reconstruction") is not yet clear to me. The authors should clarify this important point in the final version of the paper.


Positive Curvature and Hamiltonian Monte Carlo
The Jacobi metric introduced in mathematical physics can be used to analyze Hamiltonian Monte Carlo (HMC). In a geometrical setting, each step of HMC corresponds to a geodesic on a Riemannian manifold with a Jacobi metric. Our calculation of the sectional curvature of this HMC manifold allows us to see that it is positive in cases such as sampling from a high dimensional multivariate Gaussian. We show that positive curvature can be used to prove theoretical concentration results for HMC Markov chains.

This paper seem to have 3 main components:1. The authors make quite a nice general setup for relating Ollivier's notion of curvature to some ideas in differential geometry.2. The authors derive a straightforward application of Ollivier/Joulin's concentration result.3. The authors do some careful computations getting very nice analytic answers for an important simple example.The first seems quite important for bringing this to a general audience; the second is trivial but nice to write down; the third seems very important for showing that their approach really works (and proving that HMC `works well' for at least one example).Since HMC is still not well understood even for simple examples, I welcome this theoretical contribution. That said, there is one technical issues which vexes me; I am not even sure if the proposed methodology is sound because of this. The issue is that the authors estimate the minimum of curvatures from simulations. I am extremely unclear on how they are taking the "empirical infimum" of curvatures to get a lower bound on curvature and then apply bounds of Ollivier and Joulin. The bounds that they cite require the actual infimum, and it isn't at all clear to me that this empirical thing makes any sense. This sort of thing can certainly be resolved, but I don't know of any trivial way to do it, and the authors seem to gloss over the entire issue as far as I can see. They don't give even any intuition on why this must work. Minor quibbles:1. Remark 2.7 and Remark 3.3: In both of these remarks, the authors discuss calculating the curvature of the whole Markov chain by running the chain for a while and taking the sample infimum. This has a whole host of obvious problems. To mention a few: A. calculating an infimum via Monte Carlo is always dangerous in the absence of regularity assumptions, B. Calculating ANY measurement of mixing by running MCMC is even more dangerous, even with regularity assumptions, since the relevant chain may simply not have mixed. Of course, one can be lucky and everything can work out, but this seems like it careless to write down such a thing without any justification. Perhaps I am missing something here...2. They say near the bottom of pp. 5 that $\kappa \approx \frac{ Inf Sec}{6}$. In Lemma 4.1, the sectional curvatures are random variables. How does one go from the distribution of a random variable to an infimum in this setting? How does this relate to the simulation at the end of the paper?3. A few typos are scattered through out the paper; these must be corrected e.g., Equality 2.2: Sec(u,v) should be Sec_{x}(u,v)


Discovering Structure in High-Dimensional Data Through Correlation Explanation
We introduce a method to learn a hierarchy of successively more abstract representations of complex data based on optimizing an information-theoretic objective. Intuitively, the optimization searches for a set of latent factors that best explain the correlations in the data as measured by multivariate mutual information. The method is unsupervised, requires no model assumptions, and scales linearly with the number of variables which makes it an attractive approach for very high dimensional systems. We demonstrate that Correlation Explanation (CorEx) automatically discovers meaningful structure for data from diverse sources including personality tests, DNA, and human language.

This paper presents an unsupervised dimensionality reduction algorithm which is based on the information bottleneck (IB) method. The method optimizes a constrained objective which, like the IB method, is comprised of the mutual information criteria. The criteria are between a joint density of discrete observed variables and the densities of a set of discrete latent factors, and between factored densities of the observed variables and the latent factors' densities. The goal of the the objective function is to infer latent factors such that by conditioning on them, the observed variables can be factorized into subsets of minimally correlated elements. An iterative algorithm is derived for optimizing the objective which alternates between updating conditional densities over the latent factors given a set of observed training data (using fixed-point equations) and updating the assignments of the observed variables to one of the latent factors. In the numerical experiments the model is applied to learn latent tree structures on a number of benchmarks. The hierarchical structures are learned by stacking multiple instances of the model such that maximum a posteriori latent interpretations of a trained model become the observed variables for the next model instance on top.The paper presents a simple method for unsupervised dimensionality reduction which scales linearly w.r.t. the number and cardinality of variables involved. The objective function it optimizes allows for an automatic selection of the number of latent factors to be employed. Experimental results on synthetic data as well as on benchmarks from different domains show that the stacked implementation of the method performs a decent job of learning plausible/close to ground-truth latent hierarchies as compared to the other methods. On the other hand, the work lacks details about the consistency of the proposed method. The model uses fixed-point updates for estimating posteriors over the latents which are set randomly at the beginning. Moreover, the values of alpha_{i,j} are also randomly initialized (violating constraints on them) and are then gradually updated using lambda and adaptive gamma_{i,j} parameters. To that end however, no analysis is performed on model's convergence behaviour e.g., for increasingly large number and cardinalities of model variables. Also details are lacking on the sampling of training data and it is not clear how much a chosen sampling regime may effect the quality of probability estimates and in turn overall parameter evolution and performance. The method is proposed for discrete random variables and it also imposes the condition that each of the observed variables belongs to one latent factor, hence claiming that the model requires no assumptions is not actually true.For the relevant work, how does the proposed method can be contrasted with the recent work of Song et al. ICML 2013, on hierarchical decomposition of latent tree structures?Altogether the paper is well written and it is clear about conveying the main message and results.


Sparse Random Features Algorithm as Coordinate Descent in Hilbert Space
In this paper, we propose a Sparse Random Features algorithm, which learns a sparse non-linear predictor by minimizing an ℓ 1 -regularized objective function over the Hilbert Space induced from a kernel function. By interpreting the algorithm as Randomized Coordinate Descent in an infinite-dimensional space, we show the proposed approach converges to a solution within ϵ-precision of that using an exact kernel method, by drawing O(1/ϵ) random features, in contrast to the O(1/ϵ 2 ) convergence achieved by current Monte-Carlo analyses of Random Features. In our experiments, the Sparse Random Feature algorithm obtains a sparse solution that requires less memory and prediction time, while maintaining comparable performance on regression and classification tasks. Moreover, as an approximate solver for the infinite-dimensional ℓ 1 -regularized problem, the randomized approach also enjoys better convergence guarantees than a Boosting approach in the setting where the greedy Boosting step cannot be performed exactly.

The paper presents a random feature based approximation algorithm for solving the l1 regularized problem in a (possibly infinite dimensinal) Hilbert space. 		    The paper is well written and easy to read. Theorem 2 and its corollaries are interesting and form the key technical contribution. 		    Solving l1 regularized problem in Hilbert space was considered ealier (for eg. [1*], which should perhaps be cited). However the proposed random feature algorithm and more importantly, its theoretical analysis are new and non-trivial. 		    Comments:		    1. Prior work on solving l1 regularized problem in Hilbert spaces perhaps need to be summarized and cited. For e.g. [1*]. Also, alternatives like [2*] may be appropriately discussed.		    2. The empirical section may be strengthened by comparing with [1*].		    3. The current choices of \lamda seem arbitrary. Why not cross-validate for accuracy for all algorithms and report support-vectors/sparsity too? I think the current choice makes the values in the plots un-comparable. Or alternatively, provided plots with varying \lambda.		    4. Some discussion on the results is needed, for e.g, performance in case of regression seems to be better than classification. why? etc.	    [1*]. S. Rosset et.al. l1 regularization ininfinite dimensional feature spaces. COLT-2007.	    [2*]. G. Song et.al. Reproducing kernel banach spaces with the l1 norm. Journal of Applied and Computational Harmonic Analysis.


Estimation with Norm Regularization
Analysis of non-asymptotic estimation error and structured statistical recovery based on norm regularized regression, such as Lasso, needs to consider four aspects: the norm, the loss function, the design matrix, and the noise model. This paper presents generalizations of such estimation error analysis on all four aspects. We characterize the restricted error set, establish relations between error sets for the constrained and regularized problems, and present an estimation error bound applicable to any norm. Precise characterizations of the bound is presented for a variety of noise models, design matrices, including sub-Gaussian, anisotropic, and dependent samples, and loss functions, including least squares and generalized linear models. Gaussian width, a geometric measure of size of sets, and associated tools play a key role in our generalized analysis.

1) Please give examples of what new regularizers are included in your analysis. What do you cover that is not decomposable, yet yields fast estimation rates? 2) You give several generalizations of Gordon's inequality to correlated Gaussian and subGaussian design. These are only useful if we can compute the Gaussian width of the set. The few sets that we do know the Gaussian width for are sparse vectors or the similar error cone from Bickel et al. There are several examples in [6] that aren't decomposable and you may want to emphasize these norms, since they have known gaussian widths and aren't decomposable.3) Without specific examples of error cones, and their Gaussian widths, and the regularizers, the importance of the results here are not clear. For example, the regularizers covered in this work are the same as those in [6], which is broader than those in Negahban et al. Do you obtain the same rates of convergence for the Lagrangian formulation as the constrained formulation? This is expected, but it would be nice to see a few examples.4) I just came across a paper http://arxiv.org/pdf/1405.1102.pdf , and I think in Theorem 6.3 on page 11 it contains essentially the same results for "Independent anistotropic sub-gaussian designs". The theorem is stated for sets that are cone intersected with sphere, but of course any closed subset of the sphere can be represented like that. Thus it also applies to all subsets of the sphere. It may not be fair to compare against this paper since it was posted only a month before the NIPS submission deadline. On the other hand, the proof in the Tropp paper seems really easy. Can you please comment?The paper by Mendelson et al. 2007 is also relevant, http://link.springer.com/article/10.1007%2Fs00039-007-0618-7Please discuss the relationship to these two papers.


Decoupled Variational Gaussian Inference
Variational Gaussian (VG) inference methods that optimize a lower bound to the marginal likelihood are a popular approach for Bayesian inference. A difficulty remains in computation of the lower bound when the latent dimensionality L is large. Even though the lower bound is concave for many models, its computation requires optimization over O(L 2 ) variational parameters. Efficient reparameterization schemes can reduce the number of parameters, but give inaccurate solutions or destroy concavity leading to slow convergence. We propose decoupled variational inference that brings the best of both worlds together. First, it maximizes a Lagrangian of the lower bound reducing the number of parameters to O(N ), where N is the number of data examples. The reparameterization obtained is unique and recovers maxima of the lower-bound even when it is not concave. Second, our method maximizes the lower bound using a sequence of convex problems, each of which is parallellizable over data examples. Each gradient computation reduces to prediction in a pseudo linear regression model, thereby avoiding all direct computations of the covariance and only requiring its linear projections. Theoretically, our method converges at the same rate as existing methods in the case of concave lower bounds, while remaining convergent at a reasonable rate for the non-concave case.

The paper presents new techniques, and theoretical results relating to those techniques, to optimise the Variational Gaussian lower bound to the log evidence in latent Gaussian models LGMs. The technique could be applied to a broader category of latent linear models but their presentation focuses on LGMs only.VG approximate inference is important because it has many nice properties : it is widely applicable, often quite accurate and relatively fast.This paper attempts to makes VG methods more scalable without resorting to making factorisation assumptions on the approximating Gaussian distribution.Whilst the authors show how the objective function can be `decoupled' they do not show experimentally that this leads to a clear improvement in speed or scalability over standard techniques. Their largest problem has only 1.5k data points. I can only imagine that this because there is some scalability issue in their approach/implementation.Since the covariance of the VG approximation is not explicitly maintained i think it is critical that the authors explain to readers how expectations with respect to this density can be calculated once their solver has found the optimum. We are not just interested in the value of the bound but the moments of the approximating Gaussian also.The derivations of the theoretical results should be presented in full in the supplementary material. The theoretical results are quite difficult to follow. Whilst their theorectical contributions are interesting I wonder how useful they are -- solving a lagrangian with non-linear constraints is difficult. Their linearisation method is not guaranteed to converge.Furthermore, one of the nice things about Seeger's local variational bounding method is that you could specify the rank of the covariance approximation. Could you do that in this context? Or is the entire covariance always approximated -- in which case how scalable is this method anyway. QualityThe experiments could be more convincing. This paper is about making VG inference scalable and fast? The largest problem to which their method is applied has only 1500 data points -- I wouldn't classify this as large. At this scale they only achieve a marginal improvement over the Cholesky method. The experiments do not detail the stopping conditions used on either algorithm and so making comparisons is difficult. Also, it is difficult to see what numeric bound values were achieved for either method. I think these results would be better presented in a table, with means and standard errors over multiple runs.It is especially important that their experiments show the results obtained over multiple runs since their solver is not guaranteed to converge.ClarityThe authors need to spend a bit more time and effort making the presentation cleaner.I think the derivations should be presented in full in the supplementary material taking time to describe the notation used and steps taken. As it stands the descriptions of the derivations in the main body of the paper are too short to understand.OriginalityThis paper presents novel results about nature of the VG bound optimisation problem. However, the usefulness of these results is unclear -- how fast would the solvers that are guaranteed to converge take in practice? If you only use a solver that isnt guaranteed to converge what is the importance of these results.SignificanceThis paper presents some novel results about the VG approximate inference optimisation problem. Minor points and typos.line 45 : Maybe Gaussian processes are not the best example of models with strong correlations between variables in this context since we would only attempt approximate inference for GP models when O(D^2) space and O(D^3) time is feasible and so wouldnt need a distributed solver.* Section 2 para 1. I appreciate that it is convenient to focus on just LGMs in this paper but i think readers would appreciate a more precise description of the broader class of models that your method could be applied to.Eq. (5) maybe explain the log det V notation.line 104: expand with no sline 134: it doesnt make sense to introduce auxiliary variables such that h_n = \bar{m_n}. What are the constraints for then? Either you introduce the auxiliary variables and the constraints or you just set h_n to be defined as \bar{m}_n. line 141: concave and convex mix up? should it not say ...function f (concave with respect to Cholesky of V).eq(6) Maybe say subject to constraints ... for n=1,..,N.Theorem 4.1 eq 8 -- you should explain that \Lambda^* = \diag(\lambda^*) .line 159. missing n subscript on \lambda^*eq (9) : i think notation used for the partial derivatives is a little unclear. You either need to explain the notation used or write the partial derivate out explicitly with a vertical "evaluated at" bar with subscript h=h^*  line 169 : the condintions that paragraph 1 page 4 : maybe change each of the "The x part..." to "Part x ..."line 177 : constrain ", line 177 : this sentence is weird ", and similar to [9]"line 179: "while the second equality is simply change of variables" line 190: there are two prerequisites...lines 194 to 202 : This is hard to follow. I would recommend taking your time, going through each of the steps explicitly and precisely defining terms in the supplementary material.line 211 : this sentence is poor.line 219 : "this optimisation appears..." which optimisation? Both formulations are difficult?line 229 : "and is an effective for " method?line 254 : "for primal function" for the primal function?line 260 : missing super script k on f ? line 266 : of the dual variables are equalline 285 : two eithers.line 291 : see Algorithm 1?line 293 : quadratic convergence under what conditions. The next sentence is missing a "this" it is also unclear.line 294: we never observe this in line 312 : Sentence starting "Basically, ..." What is the justification for this heuristic?Bibliography : Authors in references need to be made consistent


Hardness of parameter estimation in graphical models
We consider the problem of learning the canonical parameters specifying an undirected graphical model (Markov random field) from the mean parameters. For graphical models representing a minimal exponential family, the canonical parameters are uniquely determined by the mean parameters, so the problem is feasible in principle. The goal of this paper is to investigate the computational feasibility of this statistical task. Our main result shows that parameter estimation is in general intractable: no algorithm can learn the canonical parameters of a generic pair-wise binary graphical model from the mean parameters in time bounded by a polynomial in the number of variables (unless RP = NP). Indeed, such a result has been believed to be true (see [1]) but no proof was known. Our proof gives a polynomial time reduction from approximating the partition function of the hard-core model, known to be hard, to learning approximate parameters. Our reduction entails showing that the marginal polytope boundary has an inherent repulsive property, which validates an optimization procedure over the polytope that does not use any knowledge of its structure (as required by the ellipsoid method and others).

This paper studies the worst case hardness of estimating the parameters of abinary pairwise undirected graphical model. By considering the specific case ofthe hard-core / independent set model and relying on the known result thatapproximating the partition function for this problem given the parameters (evenfor the "unweighted" case of all theta_i being 0) is hard, the authors show thatthe other direction -- namely, approximating the parameters given the nodemarginals -- is hard, in the sense that it does not admit an FPRAS.This is a strong theoretical paper, addressing a computational complexityproblem that occurs very often in theory and practice, has been conjectured tobe hard, but hadn't yet been shown formally to be hard (to the best of my andthe authors' knowledge). The work is unlikely to have much impact in practice.Nonetheless, it provides new insights into the parameter estimation problemthrough the structure of their reduction, which essentially says that if onecould efficiently approximate the parameters given the marginals, then one coulduse the projected gradient method to efficiently estimate the marginals giventhe parameters (in particular, when theta_i = 0), which in turn is known to behard.The technique used in the proof, specifically that it suffices to activelyproject on to a quadrant and yet indirectly achieve projection onto the shrunkenpolytope M_1, is the main novelty. The overall reasoning seems correct: I couldnot identify any significant flaws, however I have not verified the details ofall proofs, especially those in the supplementary material.The paper is written well and often provides abstract summaries of the technicalstatements/claims to follow. One aspect that could improve the paper (or perhapsis already there but I missed) is a clear intuitive description of thereduction, along the lines that one can efficiently approximate thehard-to-approximate \theta -> \mu problem (specifically for \theta = 0) if onehad an efficient approximation to the \mu -> \theta problem, using anoptimization method that is guaranteed to involve only a polynomial number ofsteps.Minor comments:- Depending on the background of the readers, they may be very familiar with one  of 'hard-code model' and 'independent set' problems, but likely not with both.  Clarifying early on that both are the same (where for the independent set  problem one associates a weight lambda_i with each node, which then induces a  weight on every independent set) would help such readers.- In Theorem 3.1, did you mean to write lambda = 1 and d >= 6 (rather than 5)?  Otherwise it seems that lambda will be less than (d-1)^{d-1} / (d-2)^d.- In several places, inline math expressions of the form abc/def are ambiguous.  I believe in most places you mean abc/(def) rather than (abc/d) * ef. It would  be better to remove than ambiguity.- Page 3, in the top-third: in what sense is the marginal polytop M a polytope  "equal to the convex hull of independent set vectors I(G)"?- Definition 2.1: missing "y" after (1+delta).- Corollary 3.2: "degree most d" -> "degree at most d"


Self-Paced Learning with Diversity
Self-paced learning (SPL) is a recently proposed learning regime inspired by the learning process of humans and animals that gradually incorporates easy to more complex samples into training. Existing methods are limited in that they ignore an important aspect in learning: diversity. To incorporate this information, we propose an approach called self-paced learning with diversity (SPLD) which formalizes the preference for both easy and diverse samples into a general regularizer. This regularization term is independent of the learning objective, and thus can be easily generalized into various learning tasks. Albeit non-convex, the optimization of the variables included in this SPLD regularization term for sample selection can be globally solved in linearithmic time. We demonstrate that our method significantly outperforms the conventional SPL on three real-world datasets. Specifically, SPLD achieves the best MAP so far reported in literature on the Hollywood2 and Olympic Sports datasets.

In this paper, the authors introduce a formalism for self-paced learning with diversity. This is a type of “curriculum learning.” Curriculum learning, inspired by the way humans and animals appear to learn, suggests learning training examples in a particular order, for example attempting to fit easy examples prior to hard ones. One challenge is how to decide what constitutes an easy or hard example. Self-paced learning showed that you can use the loss function you are trying to minimize as a measure of difficulty. By weighting the samples you can focus on learning easier ones first. Learning proceeds by minimizing an objective function which has two terms: the weighted loss function over the samples and the negative of the L_1 norm of the weights. A parameter lambda controls the importance of the two terms and is increased while fitting to force the model to incorporate increasingly difficult examples. Because this function is biconvex it is possible to optimize relatively efficiently.This work extends self-paced learning by incorporating “diversity.” That is, the model should learn not just easy examples, but diverse examples. To achieve this, the authors assume that examples are already grouped in some way (for example, using a clustering algorithm). They propose adding a third term to the objective function, which favors weighting samples from multiple groups (and an additional parameter, gamma, to weight the importance of this term). The objective function is no longer biconvex, however, they show a simple algorithm which they prove is able to find the global minimum of the latent weights in linear time.The submission then compares the results of objective functions learned using various methods (including self-paced learning) on three datasets: multimedia event detection and two action recognition datasets. Learning with diversity outperformed self-paced learning (and other methods) for all datasets, including apparently fitting in fewer iterations.This submission builds on previous work to construct a novel approach to curriculum learning. It demonstrate that diversity may be an important property to consider in achieving good fits. It provides a well-constructed rational for the approach, a novel algorithm including proof of optimality and several well-constructed empirical tests. The approach results in significant improvements in several of the datasets considered, which are quite challenging tasks. The authors propose several possible extensions to their approach for further research. Additionally, the use of “diversity” during learning may have some generality, and may be applied in other contexts.The manuscript is well presented and does an excellent job of conveying the key ideas, while including proofs and additional detail in the supplementary.One weakness of the manuscript, is that it lacks empirical comparisons with some more known tasks and datasets (for example, MNIST, as in the original SPL work). The datasets chosen here both have obvious diversity (examples from the same category that are very different) and levels of difficulty (examples which are obviously more ambiguous than others). This makes them good choices for testing the ideas presented. However, it would have been helpful to examine well-studied datasets such as MNIST which do not necessarily have the same obvious levels of diversity and difficulty. Does this approach still result in improved fits? It is not obvious to me what the answer would be and it would motivate helpful discussion about the scope of this work.


Feature Cross-Substitution in Adversarial Classification
The success of machine learning, particularly in supervised settings, has led to numerous attempts to apply it in adversarial settings such as spam and malware detection. The core challenge in this class of applications is that adversaries are not static data generators, but make a deliberate effort to evade the classifiers deployed to detect them. We investigate both the problem of modeling the objectives of such adversaries, as well as the algorithmic problem of accounting for rational, objective-driven adversaries. In particular, we demonstrate severe shortcomings of feature reduction in adversarial settings using several natural adversarial objective functions, an observation that is particularly pronounced when the adversary is able to substitute across similar features (for example, replace words with synonyms or replace letters in words). We offer a simple heuristic method for making learning more robust to feature cross-substitution attacks. We then present a more general approach based on mixed-integer linear programming with constraint generation, which implicitly trades off overfitting and feature selection in an adversarial setting using a sparse regularizer along with an evasion model. Our approach is the first method for combining an adversarial classification algorithm with a very general class of models of adversarial classifier evasion. We show that our algorithmic approach significantly outperforms state-of-the-art alternatives.

This paper tackles the problem of classification in an adversarial setting, specifically in the case of spam and phishing detection.  It presents a class of adversaries that use "feature cross-substitution" to evade classification while still accomplishing their attacks as well as two approaches to handling this type of adversary: a heuristic one based on feature equivalence classes and a more principled one based on a Stackleberg game solved with integer/linear programming.Positive Points:1.  The paper is well-written and well-motivated, with a good coverage of the related literature.  One possible bit of related work that might be included is that of Karlberger (et al.?) on Penetrating Bayesian Spam Filters.2. The idea of a complementary form of classification error (c.f. bias and variance) in adversarial learning settings is nice.  Probably not an earth shatteringly new/surprising idea, but it is nice to see people thinking about this kind of thing.3. Empirical results on three different email data sets, compared to state-of-the-art methods, are impressive.Points of Concern:4. In Sec. 4, is the simple model of 1- and 2-letter substitutions a reasonable approximation for more sophisticated types of substitution attacks?5. In Sec. 4, the first key observation, that "when the adversary uses either substitution cost function, the optimal number of features increases", while intuitively appealing, doesn't seem to me to be demonstrated very clearly at all in the reported experiments.  The claimed "right shift" does not seem obvious to me in the majority of the results shown in Fig. 2.  Indeed, if I were looking at these data in an unbiased way, I'd be tempted to say they show that in fact there is no shift in the majority of cases and, therefore, there is no noticeable tradeoff between overfitting and feature selection under adversarial conditions.  Which in turns lead to the question of whether this proposed third type of error really exists.  The rest of the paper is pretty clear and convincing (and the empirical results certainly seem to show that something is going on) -- am I missing something here?6. In Sec. 4, the second key observation seems obvious on the one hand---of course evasion will account for a large amount of error if there is a lot of malicious activity---and, at the same time, I wonder if we can accurately make this claim; that is, are we confident that the bias and variance components of the error really are small under this scenario? 7. Seems like the last paragraph of Sec. 6 is hiding lots and lots and lots of details.  I realize NIPS has a tight page limit, but it seems like a bit more here, traded with a bit less somewhere else would make this a better paper.Small points:8. References could be tidied up, particularly capitalization issues.9. In Sec. 2, subheading "The Game", should "the defender" in bullet 3 be reconciled with the "learner" in bullet 1?10. I'm not clear on what exactly the notation $x^A_j \oplus x^'_j$ means.11. In Fig. 1, right hand side, does the x-axis label "# Features" refer to the size of the equivalence class (it seems to be hinted that it does in the text, but this should be clarified).12. There are a few places throughout the paper where there are usage issues with articles (mostly they are occasionally missing).13. In Sec. 4, when details on the data sets are given, class balance is described for both the Enron and UCI data sets but is neglected for the Ling data set.14. In Sec. 6, I had a hard time with the definition, $e_j = \Sum…$, where $e_j = w^T…$.  Is this just an odd way to say how we are going to compute the $w^TA()$ term in Eq. 4? Or, is something else being said here (with a possible typo obfuscating it)?   Can this be clarified somehow?Interesting questions:15. With a large enough cost budget, it would seem a foregone conclusion that the defender will have little hope of accurately identifying spam while also allowing legitimate email through.  However, it would also seem that at some threshold of substitution, spam effectiveness will begin to really suffer (that is, not even really gullible people will give it any credence at some point).  This could be modeled with a non-linear cost function or with a simple threshold, for example, but do we have any way to bound this effective cost budget?  That is, is there a point at which we don't have to worry about detecting it because no adversary would choose such a highly substituted attack vector?


Fast Prediction for Large-Scale Kernel Machines
Kernel machines such as kernel SVM and kernel ridge regression usually construct high quality models; however, their use in real-world applications remains limited due to the high prediction cost. In this paper, we present two novel insights for improving the prediction efficiency of kernel machines. First, we show that by adding "pseudo landmark points" to the classical Nyström kernel approximation in an elegant way, we can significantly reduce the prediction error without much additional prediction cost. Second, we provide a new theoretical analysis on bounding the error of the solution computed by using Nyström kernel approximation method, and show that the error is related to the weighted kmeans objective function where the weights are given by the model computed from the original kernel. This theoretical insight suggests a new landmark point selection technique for the situation where we have knowledge of the original model. Based on these two insights, we provide a divide-and-conquer framework for improving the prediction speed. First, we divide the whole problem into smaller local subproblems to reduce the problem size. In the second phase, we develop a kernel approximation based fast prediction approach within each subproblem. We apply our algorithm to real world large-scale classification and regression datasets, and show that the proposed algorithm is consistently and significantly better than other competitors. For example, on the Covertype classification problem, in terms of prediction time, our algorithm achieves more than 10000 times speedup over the full kernel SVM, and a two-fold speedup over the state-of-the-art LDKL approach , while obtaining much higher prediction accuracy than LDKL (95.2% vs. 89.53%).

The paper addresses the problem of speeding up prediction of kernel methods for large-scale problems for which the models are too large and hence the prediction is too slow. The proposed approach entails construction of pseudo landmark points, in the spirit of Nystrom approximation. Unlike the true landmark points which are sampled from the data, for pseudo landmark points the values of their kernel dot products with arbitrary points can be predicted from the same values for the landmark points. Hence the increased accuracy of model approximation can be attained with the same computational complexity as using the original set of landmark points. While the specific idea proposed in the paper is certainly novel, the overall conceptual motivation for the paper is somewhat weak in my opinion. Especially for large-scale problems, it is not the prediction but the training phase that constitutes the main computational bottleneck. It is virtually unfeasible to train large-scale models with nonlinear kernels when the kernel matrix must be stored in memory. Assuming this problem can be solved by low-rank approximation the complexity of the resulting model would then be tractable, hence I see no major gain to be made from speeding up the prediction phase. The idea of pseudo landmark points may be interesting to increase the prediction accuracy at a fixed complexity of the training model. The goal would then be not to accurately approximate the kernel matrix but rather to raise the predictive ability of the model (at fixed cost). However, the analysis provided in the paper does not go into this direction. On the technical part, several issues are unclear:1. No bias is considered in the SVM formulation in line 111. 2. Eq. 2: Why is G used in the definition of \bar{W}? This is counter-intuitive: we are trying to approximate G with \bar{G} depending on \bar{W} depending on G. A typo? 3. What is measured in Table 1? What is the meaning of 'x' in the values like '12.8x' etc? This table is not explained very well in the ensuing text. One can guess that the numbers pertain to some speedup, but what is the baseline for this comparison?  UPDATE:The notation used in the paper you have referred to in the response is much more clear, and computational implications are clearly explained. Please update your notation accordingly for the sake of self-sufficiency. The 'x' notation should be clearly expalained: I have almost misinterpreted it again when reading your response.


Deep Learning Face Representation by Joint Identification-Verification
The key challenge of face recognition is to develop effective feature representations for reducing intra-personal variations while enlarging inter-personal differences. In this paper, we show that it can be well solved with deep learning and using both face identification and verification signals as supervision. The Deep IDentification-verification features (DeepID2) are learned with carefully designed deep convolutional networks. The face identification task increases the inter-personal variations by drawing DeepID2 features extracted from different identities apart, while the face verification task reduces the intra-personal variations by pulling DeepID2 features extracted from the same identity together, both of which are essential to face recognition. The learned DeepID2 features can be well generalized to new identities unseen in the training data. On the challenging LFW dataset [11], 99.15% face verification accuracy is achieved. Compared with the best previous deep learning result [20] on LFW, the error rate has been significantly reduced by 67%.

This paper's core finding is that combining an identity classification task as well as metric-learning-style verification task helps to learn better features for face classification/verification.  The "verification task" here tries to decrease feature-space distance between instances of the same identity, and increase distance between those of different identities.  This improvement is embedded in a state-of-the-art system for face verification, which uses convnets trained on many (400) different views to generate features, distilled into a small set of 25 using feature selection.  Very good results are obtained and experiments performed using LFW as a test set.Overall, these are very good results obtained using a somewhat complex pipeline, and a good investigation into the contribution of each "task" in the loss for feature learning.  Combining the tasks has been hinted at before (e.g. siamese network after classification pre-training for DeepFace), but this work brings this out specifically.  Some of the writing could be a bit less dry (and edited for English), but this does not interfere with the understanding of the paper.Further comments:* I don't understand how "m" in eqn 1 was selected.  If it is learned as a parameter, it seems it would collapse to m=0, or at most the minimum ||fi-fj|| (even with the procedure indicated at l.152), since this would always give zero Verif error for negative pairs.* Is there a more concrete reason why including the identification task improves verification performance and/or learning ("richer information" is a bit vague)?  E.g. the gradient from the verification task pushes apart just two points, while the softmax grad in the identification task pushes the correct identity away from the classification-layer templates for *all* other identities at the same time.* It would be nice to see individual performance of the best few selected network regions/views, or an evaluation using 1, 2, 4, ..., 25 views.  The final combination achieves good performance, but it would be good to also see the individual contributions.* l.38 "eternal topic":  this is an odd choice of word (it implies the topic will be debated forever, when in fact it seems to be making good progress); a "central topic" may be better here.