VARIANCE NETWORKS: WHEN EXPECTATION DOES NOT MEET YOUR EXPECTATIONS
Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. It means that each object is represented by a zero-mean distribution in the space of the activations. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned.
This paper studies variance neural networks, which approximate the posterior of Bayesian neural networks with zero-mean Gaussian distributions. The inference results are surprisingly well though there is no information in the mean of the posterior. It further shows that the several variational dropout methods are closed related to the proposed method. The experiment indicates that the ELBO can actually better optimized with this restricted form of variational distribution. 
The paper is clearly written and easy to follow. The technique in the paper is solid.
However, the authors might need to clarify a few questions below. 
Q1:  if every transformation is antisymmetric non-linear, then it seems that the expected distribution of $t$ in (2) is zero. Is this true or not? In another word, class information has to be read out from the encoding of instances in Fig 1. It seems antisymmetric operators cannot do so, as it will only get symmetric distributions from symmetric distributions. 
Q2: it is not straightforward to see why KL term needs to go zero. In my understanding, the posterior aims to fit two objectives: maximizing data likelihood and minimizing KL term. When the signal from the data is strong (e.g. large amount of data), the first objective becomes more important. Then q does not really try to make KL zero, and alpha has no reason to go infinity. Can you explain more? 
Q3: Is the claimed benefit from the optimization procedure or the special structure of the variance layer? Is it possible to test the hypothesis by 1) initializing a q distribution with learnable mean by the solution of variance neural network and then 2) optimizing q? Then the optimization procedure should continue to increase ELBO. Then compare the learned q against the variance neural network. If the learned q is better than the variance network -- it means the network structure is better for optimization, but the structure itself might not be so special. If the learned q is worse than the variance network, then the structure is interesting. 
A few detailed comments:
1. logU used without definition. 
2. if the paper has a few sentence explaining "Gaussian dropout approximate posterior", section 4 will be smoother to read.