In many applications we seek to optimize an expectation with respect to a distribution over discrete variables. Estimating gradients of such objectives with respect to the distribution parameters is a challenging problem. We analyze existing solutions including finite-difference (FD) estimators and continuous relaxation (CR) estimators in terms of bias and variance. We show that the commonly used Gumbel-Softmax estimator is biased and propose a simple method to reduce it. We also derive a simpler piece-wise linear continuous relaxation that also possesses reduced bias. We demonstrate empirically that reduced bias leads to a better performance in variational inference and on binary optimization tasks.
The papers studies estimators of gradients taken from expectations with respect to the distribution parameters. The paper has studied two main types of estimators, Finite Difference and Continuous Relaxation. The paper made several improvements to existing estimators. 
My rating of the paper in different aspects (quality 6, clarity 8, originality 6, significance 4). 
Pros: 
1. The paper has made a nice introduction of FD and CR estimators. The improvements over previous estimators are concrete -- it is generally clear to see the benefit of these improvements. 
2. The first method reduces the running time of the RAM estimator. The second method (IGM) reduces the bias of GM estimator. The first improvement avoids many function evaluations when the probability is extreme. The second improvement helps to correct bias introduced by continuous approximation of \zeta_i itself. 
Cons: 
1. the paper content is a little disjointed: the improvement over RAM has not much relation with later improvements. It seems the paper is stacking different things into the paper. 
2. All these improvements are not very significant considering a few previous papers on this topic. Some arguments are not rigorous. (see details below)
3. A few important papers are not well discussed and omitted from the experiment section. 
Detailed comments
1. The REBAR estimator [Tucker et al., 2017] and the LAX estimator [Grathwohl et al., 2018] use continuous approximation and correct it to be unbiased. These papers in this thread are not well discussed in the paper. They are not compared in the experiment either.  
2. In the equation 7 and above: what does 4 mean? When beta \neq 4, do you still get unbiased estimation? My understanding is that the estimator is unbiased only when beta=4. (correct me if I'm wrong)
3. The paper argues that the variance of the estimator is mostly decided by the variance of q(zeta)^-1 when the function is smooth. I feel this argument is not very clear. First, what do you mean by saying the function is smooth? The derivative is near a constant in [0, 1]? 
4. In the PWL development, the paper argues that we can choose alpha_i \approx 1/(q_i(1-q_i)) to minimize the variance. However, my understanding is, the smaller alpha_i, the smaller variance.