Thanks for the detailed explanation. This is very helpful.
I mainly target at evaluating models in 1-bit precision with QAT, so I guess the second approach is good enough (for now). It seems that I still will need touch the Aten/c10 code and recompile it locally. If so, is this something the PyTorch team interested to merge upstream in the future (assuming my code meets the quality requirement)?
On a related note, Nvidia’s new A100 architecture 11 will support binary (1-bit) precision.

Acceleration for all data types, including FP16, BF16, TF32, FP64, INT8, INT4, and Binary

This is not too far away from the production. Details can also be found in their white paper 12. It feel like an interesting direction for PyTorch community to explore and will be meaningful if we can support in the long-run.