I agree with the comment of sub-8-bit quantization. We should be able to support 2-7 bit using the existing infrastructure with some new data types INT2-7.
In the case of 1-bit (binary), you can represent {-1, 1} in {0, 1} by assigning -1 to 0. In fact, that’s what will be implemented in hardware. However, that means you will replace multiplication by XNOR. This change results in a separate set of operators/functionals/modules need to be overload for binary network. From math point of view, I would like to see BNN implemented in this way (which exactly match the hardware). However, it is a lot of work and hard to be maintained (separately from all other NN modules). Frankly, an engineer will argue that there is no significant benefit of doing so. I feel like a new data type BINT1 for {-1, 1} (to be different from INT1 for {0, 1}) is a better choice.
I will try to experiment with this idea and submit issue/PR in the coming months.