In many applications, it is desirable to extract only the relevant information from complex input data, which involves making a decision about which input features are relevant. The information bottleneck method formalizes this as an informationtheoretic optimization problem by maintaining an optimal tradeoff between compression (throwing away irrelevant input information), and predicting the target. In many problem settings, including the reinforcement learning problems we consider in this work, we might prefer to compress only part of the input. This is typically the case when we have a standard conditioning input, such as a state observation, and a "privileged" input, which might correspond to the goal of a task, the output of a costly planning algorithm, or communication with another agent. In such cases, we might prefer to compress the privileged input, either to achieve better generalization (e.g., with respect to goals) or to minimize access to costly information (e.g., in the case of communication). Practical implementations of the information bottleneck based on variational inference require access to the privileged input in order to compute the bottleneck variable, so although they perform compression, this compression operation itself needs unrestricted, lossless access. In this work, we propose the variational bandwidth bottleneck, which decides for each example on the estimated value of the privileged information before seeing it, i.e., only based on the standard input, and then accordingly chooses stochastically, whether to access the privileged input or not. We formulate a tractable approximation to this framework and demonstrate in a series of reinforcement learning experiments that it can improve generalization and reduce access to computationally costly information.
Summary - The paper proposes an approach, called the Variational Bandwidth Bottleneck (VBB), capable of compressing only a part of the input and still learn representations that are informative of the output. The approach is motivated from the following perspective -- there might be situations where a ``part’’ of the input is privileged in the sense that it may be costly / wasteful to maintain access to all the time, thereby rendering the standard IB pipeline infeasible (as it requires unrestricted access to the entire input). By breaking down the input into standard (always available) and privileged (not always available) components, the paper proposes a module that decides whether to access the privileged input during compression based on the standard input. The goal is to be able to decide when to access privileged information and not how to break the overall input into standard and privileged components. The approach tackles a narrow subset of problems compared to the standard information bottleneck. The authors show the applicability of the proposed approach in reinforcement learning setups -- specifically, (1) when to access an expensive model-based planner for goal-driven navigation; (2) when to access goal-information in goal-driven navigation and (3) treating communication in a multi-agent cooperative setting as ``privileged’’ information. The experimental results demonstrate that VBB accesses privileged information in a feasible and minimal manner and results in better generalization performance.
Strengths
- Apart from the flaws (highlighted in weaknesses), the paper is well-written and generally easy to follow.
- The problem statement is well-motivated. The authors did a good job of first identifying when the standard IB approach would be infeasible / costly -- even though representations are compressed and relevant, computing them still requires unrestricted access to input -- and then motivating the need for selective access to information while decision-making. The problem is well-grounded in the experimental settings the authors provide results for.
- Apart from the concerns (highlighted in weaknesses), the experimental results generally support the claims of the paper. Results in Sec. 7.1 (loosely) demonstrate that VBB accesses privileged information states where degree of freedom in terms of possible trajectories is higher -- this result, although not explored in it’s entirety, also correlates with notions of decision-states (as pointed out by authors) and bottleneck states in the existing literature. Results in Sec. 7.2 demonstrate improved generalization performance in terms of transfer. Additionally, results in Table.3 suggest that VBB accesses privileged information the least number of times. Similarly, results in Sec. 7.3 indicate VBB results in improvements in performance in the multi-agent setting while resulting in minimal communication among the agents. The baselines being compared with in the paper are also reasonable.
- The problem statement and the proposed approach have some degree of novelty. Most works along the lines of restricting access to relevant information still assume unrestricted access to the `entire’ information. 
Weaknesses
That being said, the paper does have some flaws / clarity issues that make several associated details confusing when judged in context of prior work. These concerns (in addition to the strengths highlighted above) mostly surrounding experiments form the basis of my rating and addressing these would definitely make the paper stronger.
- One of the underlying motivations / intuitions behind restricting access to the privileged information is that fact that -- “..avoid accessing the privileged input because we want to generalize with respect to it..”. This statement in isolation is misleading and makes things unclear. From a complete generalization standpoint, preventing the encoder from overfitting to the input (‘privileged’ or ‘standard’) can be addressed by just tracking the relevance (predictive performance) based on the representations on a held-out set. Unless I am missing something, generalization of the learned representations in standard IB becomes major problem only when we restrict (costly or otherwise) access to the privileged input. The statement seems to suggest that learning representations completely and generally agnostic to the privileged input is a good idea. However, this is only true when at test-time, the privileged input may be significantly different than what was seen during training time -- a different model-based planner / entirely different goal-specifications, etc. Could the authors comment more on this and reframe the intuition wherever applicable?
- Last 3 lines of the 1st paragraph in section 3 (page 2) are incomplete -- “...constraining the channel capacity .. is permitted to differ from the prior r(Z)....”. The difference in the posterior p(Z|X) and the prior r(Z) quantify the channel capacity only under expectation over the distribution over inputs -- p(X). Therefore, in practice, it is also governed by the empirical distribution of the data (see page 4 of https://arxiv.org/pdf/1612.00410.pdf). The authors should change the lines to reflect the same.
- One of the key contributions of the paper is to develop a mechanism where one isn’t required to access the privileged input all the time -- depending on the standard input, one can decide whether to access the privileged information. This manifests in form of a mixture-distribution over a dirac-delta transformation of the deterministic encoder and a prior distribution. At inference-time, the channel capacity (d_cap, based on the standard input) can be used to decide whether to access the privileged information or not. There’s lack of clarity in terms of what happens at training time -- is it the case that (1) d_cap is computed, f(s, g) is also computed and based on the KL-term in Eq (3), B(S) is incentivized to generate d_cap that results in less frequent access the privileged information (d_cap < 0.5 for the most part) or (2) d_cap is computed, a sample is drawn from the bernoulli and z is sampled from either r(Z) or \del_(f(S, G) and KL is either 0 (if z ~ r(Z)) or a finite value (if z ~  \del_(f(S, G))? (1) involves an exact computation and always requires access to G (privileged input) during training whereas (2) is approximate and does not always require access to G. It’s unclear what the pipeline is from the paper. Can the authors clarify this? Are there specific reasons why (1) / (2) was chosen?
- Experimental Results: Highlighting these below:
     - In the experimental setting of Figure 2, do the authors notice any (1) qualitative differences in terms of where the agent accesses the output of the planner when InfoBot is used (adapted to this setting) and (2) quantitative differences (Table. 2) in terms of how often does InfoBot and the other proposed baselines access the planner output at junctions and hallways? Junctions in 2D mazes can potentially be identified as important decision states using simpler approaches. As a demonstrative experiment, the key point to be made using this experiment seems to be how often does VBB access the planner output compared to InfoBot and other baselines. 
    - In Sec. 7.2, it is unclear how VBB is being used to generalize to novel environments -- as in, is the pipeline same as InfoBot, where a frozen encoder is used to provide an exploration incentive? 
    - While the generalization results in Table. 2 are impressive in terms of success values, I think it lacks a few numbers (assuming the transfer pipeline to novel environments is same as InfoBot) -- given that the environments being tested on are different from InfoBot, how well do count-based exploration and goal-conditioned A2C baselines perform? This is to understand whether InfoBot is the right thing to compare with in these environments. 
    - Furthermore, for the goal-driven navigation set of experiments can the authors report comparisons in terms of sample-efficiency as well -- how do success-rates and average task-returns vary with time-steps of training? 
    - For results in Sec. 7.3, Table. 4, I would encourage the authors to compare with IC3Net (https://arxiv.org/pdf/1812.09755.pdf) which learns “when to communicate” in a multi-agent setting irrespective of whether it’s a cooperative situation or not.
Reasons for rating
Apart from the points mentioned above, I don’t have major weaknesses to point out. The paper is generally easy to follow and the proposed approach and problem statement is well-grounded and somewhat novel. However, the paper suffers from lack of experimental details and comparisons (and other weaknesses highlighted above) and therefore I am inclined towards my current rating. Addressing those would significantly benefit the paper and help me in  increasing my score.