The difficulty there is PyTorch inherently assumes that things are at least 1 byte when doing things with memory.
I’d probably convert to TVM 9 and see what can be done there.
(QAT with fake quantization probably could work for 4 bits, too.)