Update: I fixed this issue by changing
self.weight = torch.nn.Parameter(torch.randn(out_features, in_features, dtype=torch.qint8))

to
self.weight = torch._empty_affine_quantized([out_features, in_features], scale=1.0, zero_point=0, dtype=torch.qint8))

but now I am getting a new error that says:

RuntimeError: Only Tensors of floating point and complex dtype can require gradients