Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings. However, BNNs suffer from performance degradation mainly due to the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem by reducing the discrepancy between activation functions used at forward pass and its differentiable approximation used at backward pass, which is an indirect measure. In this work, we use the gradient of smoothed loss function to better estimate the gradient mismatch in quantized neural network. Analysis using the gradient mismatch estimator indicates that using higher precision for activation is more effective than modifying the differentiable approximation of activation function. Based on the observation, we propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.
This paper proposes a new measure of gradient mismatch for training binary networks, and additionally proposes a method for getting better performance out of binary networks by initializing them to behave like a ternary network.
I found the new measure of gradient deviation fairly underdeveloped, and I suspect the method of converting ternary activations into binary activations works for a different reason than that proposed by the authors.
There were English language issues that somewhat reduced clarity, though the intended meaning was always understandable.
Detailed comments:
"Binary Neural Network (BNN) has been gaining interest thanks to its computing cost reduction and memory saving." --> "Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings." (will stop making English language corrections from here on)
"Therefore, we argue that the sharp accuracy drop for the binary activation stems from the inefficient training method, not the capacity of the model."
This could also be due to poor initialization in the binary case. e.g., it might make sense to initialize the binary network with bias=-0.5, so that the nonlinearity has a kink at pre-activation=0, rather than pre-activation=0.5.
"Unfortunately, it is not possible to measure the amount of gradient mismatch directly because the
true gradient of a quantized activation function is zero almost everywhere. " It *is* possible to measure the mismatch to the true gradient exactly. One could even train using the true gradient. It's just that the true gradient is useless.
Fig 1b -- this is a nice baseline.
"the steepest descent direction, which is the direction toward the point with the smallest loss at given distance"
This is not the usual definition of steepest descent direction. If you're going to redefine this, should do so mathematically and precisely (for instance, you are going to run into trouble with the word "distance", since your coordinate discrete gradient more closely resembles an L\infty-ball perturbation, rather than an L2-ball perturbation.
eq. 3:
Note that this equation is equivalent to taking the true gradient of a function which has been boxcar-smoothed along each parameter. This may more closely resemble existing measures of deviation than you like.
You should also consider the relationship to an evolutionary strategies style gradient estimate, which similarly provides an unbiased gradient estimate for a smoothed function, and which allows that estimate to be computed with fewer samples (at the cost of higher error).
Sec. 4.2 / Figure 3:
The results in this section will be *highly* sensitive to the choice of epsilon. You should discuss this, specify the epsilon used, and experimentally explore the dependence on epsilon.
"The results indicate that the cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better than previous approaches. "
Don't know that I followed this. Gradient mismatch is never formally defined, so it's hard to know what this says about its relationship. Additionally, CDG sounds more like something which is correlated with, rather than an explanation for, performance.
" cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better " --> " cosine similarity between coarse gradient and CDG can explain the relationship between gradient mismatch and performance of model better "
"we shift the bias of BN layer which comes right before the activation function layer. "
Did you try using these bias values without pre-training as a ternary network? I suspect it would work just as well!
"Please note that BN layers followed by binary activation layer can be merged to the threshold of the binary activation layer, incurring no overhead at inference stage."
Did not understand this.
"it is expected that the fine-tuning increases the accuracy even further"
Does it improve the accuracy further? Should state this as result, not prediction, and should have an ablation experiment showing this.
"Table 2 shows the validation accuracy of BNN in various schemes."
Why not test accuracy?
Figure 6:
What are the filled circles?
What was the sampling grid for the HP search? The images have high spatial frequency structure that I suspect is an artifact of the interpolation function, rather than in the data.
----
Update post-rebuttal:
The authors have addressed the majority of my concerns, through both text changes and significant additional experiments. I am therefore increasing my score. Thank you for your hard work!