Hi @Vasiliy_Kuznetsov , thank you! So I ended up doing something similar, which is basically casting the quantized tensors to regular float32 tensors. I just need to pass them to my own cuda backend, so just before I do, I use quantize() and int_repr() to convert them to char for the GPU to process. But storing them as float32 tensors allows me to use the nn.functional functions on them, which is a big plus for me.
Also, I’m not sure if I will need the backward pass for what I am doing but if I do, would the regular backward pass (e.g., shown here) work for that? Since all my tensors are float32 anyway and I quantize them (using the scale and zero_point which I have saved as parameters), it should be able to do a backward pass the same as a normal nn.functional.linear function, am I right?