text: Title: Benchmark each GEMM/GEMV kernels independently
  Body: Hi,
  I tried QuantLinear from qlinear_cuda under auto_gptq.nn_moduldes.qlinear but its performance is low with skinny matmul (i.e. matmul shapes at token gen).
  Its perf is even worse than fp32
  for example, when M, N, K = 8, 11008, 11008 
  
  nn.Linear perf(torch.float16 GEMM), tops=4.988040446327411, ms=0.3886931240558624, mem_bw=624.4113131394598
  auto_gptq_gemm (wbits=8) perf, tops=1.1186859903537671, ms=1.7331199645996094, mem_bw=70.12112403198577
  auto_gptq_gemm (wbits=4) perf, tops=1.079567580111752, ms=1.7959200143814087, mem_bw=33.93262924406488
  
  its triton version was tested using kernels in gptq_quant ([link](https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/cuda/README.md))
  gptq_triton_gemm (wbits=8) perf, tops=4.8018665417962785, ms=0.40376320481300354, mem_bw=300.9890910101229
  gptq_triton_gemm (wbits=4) perf, tops=5.00364444477819, ms=0.3874809741973877, mem_bw=157.27298127664005
  
  but from the blogs and the reads, these fused dequantization + vector-matmul kernels in cuda (like "VecQuant8MatMulKernel") is supposed to be better than regular fp16 kernels
  
  
  the script i used is like the following
  
  >   
      layer = nn.Linear(K, N)
      layer.requires_grad_(False)
      layer.bias.fill_(0.0)
      vec = torch.randn([M, K]).to(DEV)
      fp16_vec = vec.type(torch.float16)
      import_auto_gptq = True
  
      if import_auto_gptq:
          from auto_gptq.nn_modules.qlinear.qlinear_cuda import QuantLinear as gptq_QuantLinear
          import quant as gptq_quant
          from gptq import GPTQ, Observer
  
          name = "gptq-layer"
          groupsize = K
          act_order = True
          percdamp = .01
          sym = True
  
          gptq_layer = GPTQ(layer, observe=False)
          gptq_layer.add_batch(fp16_vec, torch_fp16_out)
          gptq_layer.quantizer.configure(wbits, perchannel=True, sym=sym, mse=False)
          auto_gptq_linear = gptq_QuantLinear(wbits, groupsize, layer.in_features, layer.out_features, layer.bias is not None)
  
          scale, zero, g_idx, error = gptq_layer.fasterquant(percdamp=percdamp, groupsize=groupsize, actorder=act_order, name=name)
          quantizer_cpu, scale_cpu, zero_cpu, g_idx_cpu = gptq_layer.quantizer.cpu(), scale.cpu(), zero.cpu(), g_idx.cpu()
          auto_gptq_linear.pack(layer_cpu, scale_cpu, zero_cpu, g_idx_cpu)
          auto_gptq_linear.to(DEV)
  
          auto_gptq_gemm_fn = lambda: auto_gptq_linear.forward(fp16_vec)
          ms, min_ms, max_ms = triton.testing.do_bench(auto_gptq_gemm_fn)
          gemm_tops = tops(ms)
          gemm_dsize = fp16_vec.nelement() * fp16_vec.element_size() + auto_gptq_linear.qweight.nelement() * auto_gptq_linear.qweight.element_size(
          ) + torch_fp16_out.nelement() * torch_fp16_out.element_size()
          mem_bw = lambda ms: (gemm_dsize* 1e-9) / (ms * 1e-3)
          perf = get_perf(auto_gptq_gemm_fn)
          print(f"auto_gptq_gemm (wbits={wbits}) perf, tops={gemm_tops}, ms={ms}, mem_bw={mem_bw(ms)}, perf={perf}")
  
  i also checked it using torch.profile like shown below
  
  ![image](https://github.com/PanQiWei/AutoGPTQ/assets/13525892/e98fb9ff-f645-48c0-a18c-db7e71fd6060)
  
  
  