Yeah, I would recommend using FX Graph Mode Quantization for this. We have post training quantization tutorial here: (prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 1.10.0+cu102 documentation 2 (we might add a QAT tutorial later). You can use prepare_qat_fx and use the qconfig_dict api to do this.
We do have a quint4x2 dtype currently: pytorch/test_quantize_fx.py at master · pytorch/pytorch · GitHub 1, although I think this is mostly for weight. To support this with activation, I think you need:

Add support for quint4x2 in quantize_per_tensor pytorch/QTensor.cpp at master · pytorch/pytorch · GitHub

Use the is_reference option during convert_fx, which will produce a model with (dequant - float_op - quant) patterns representing the model (you can take a look at Extending PyTorch Quantization to Custom Backends · pytorch/pytorch Wiki · GitHub for reasons)
lower the model to the dla you are building, this can be through fx/torchscript or any ways you prefer