text: Title: [BUG] Recent changes increase VRAM consumption
  Body: **Describe the bug**
  Somewhere between d4011d29c623e739e91b842a87fce62a38c6e538 and b4eda619d0674e9ef009702cbd538836c0861a56 the VRAM usage increased dramatically. See below.
  
  Tomorrow, I will try to find a certain commit that caused this, but now I'm just giving a heads up.
  
  **Hardware details**
  GPU: GTX 3060 12GB
  
  **Software version**
  OS: Kubuntu 23.04
  Python: 3.10.11
  CUDA: 11.7
  PyTorch: 2.0.0+cu117
  transformers: 4.28.1
  accelerate: 0.19.0
  
  **To Reproduce**
  The code to measure VRAM:
  ```python
  from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
  import torch
  from transformers import AutoTokenizer
  
  model_path = "/opt/models/vicuna-13B-1.1-GPTQ-4bit-128g"
  
  model = AutoGPTQForCausalLM.from_quantized(
      model_path,
      model_basename="vicuna-13B-1.1-GPTQ-4bit-128g.latest",
      device="cuda:0",
      use_safetensors=True,
      use_triton=True,
      quantize_config=BaseQuantizeConfig(
          bits=4,
          group_size=128,
      )
  )
  
  input_text = "auto_gptq is " * 210
  tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)
  tok_result = tokenizer(input_text, return_tensors="pt")
  gen_result = model.generate(
      input_ids=tok_result["input_ids"].to("cuda:0"),
      max_new_tokens=10
  )
  
  mem_gb = round(torch.cuda.max_memory_reserved(0) / 1000 / 1000)
  print(f"VRAM: {mem_gb}MB")
  ```
  
  With the d4011d29c623e739e91b842a87fce62a38c6e538 (older commit) it gives:
  ```
  VRAM: 9299MB
  ```
  
  With the b4eda619d0674e9ef009702cbd538836c0861a56 (newer commit) it gives:
  ```
  VRAM: 10108MB
  ```
  
  800MB increase.
  
  I have GTX 3060 12GB. On the older commit I could use 16B-4bit models with 2000 context tokens (fully on GPU). With the newest changes I can only use 1300 context tokens.
  
  **Expected behavior**
  VRAM consumption should not increase by such a large margin.
  