text: Title: Comprehensive benchmarking of AutoGPTQ; Triton vs CUDA; vs old 'ooba' GfL
  Body: Over on @Laaza's text-gen-ui PR there has been discussion of inference performance.
  
  In particular, people are concerned that recent versions of GPTQ-for-LLaMa have performed worse than the older versions. This is one reason why many people still use ooba's fork of GfL.
  
  In order to help people understand the improvements AutoGPTQ has made, and also to help @PanQiWei and @qwopqwop200 look at potential performance differences in AutoGPTQ, I have compiled a list of benchmarks.
  
  I have compared AutoGPTQ CUDA, AutoGPTQ Triton and the old GfL ooba fork with CUDA. 
  
  I've compared act-order/desc_act vs not, with and without streaming in text-gen-UI, and with/without the `fused_attn` and `fused_mlp` parameters.
  
  I've not done absolutely every possible permutation of all those params. I only ran a few tests with streaming enabled in text-gen-ui as it always performs much worse. But I think I have enough here to get a good overview.
  
  I will be posting these figures in the text-gen-ui PR thread as well.
  
  **Other benchmarks to do**
  
  To these results we could also add the latest GPTQ-for-LLaMa Triton and CUDA figures. I did already benchmark that yesterday and it compared the same or worse than AutoGPTQ. But later today I will run those benchmarks again and add them to the spreadsheet in this new format.
  
  I would also like to test with some weaker/smaller GPUs, to see how performance might vary with less GPU processing available. And I'd also like to test on some larger models, to see if there is any difference in performance delta with varying model sizes.
  
  # Benchmarks of: ooba CUDA; AutoGPTQ CUDA; AutoGPTQ Triton
  
  ## Implementations tested
  * AutoGPTQ as of https://github.com/PanQiWei/AutoGPTQ/pull/43, using text-gen-ui from [LaaZa's AutoGPTQ PR]( https://github.com/LaaZa/text-generation-webui/tree/AutoGPTQ)
  * GPTQ-for-LLaMa using [ooba's GfL fork (oobabooga/GPTQ-for-LLaMa)](https://github.com/oobabooga/GPTQ-for-LLaMa), using text-gen-ui from `main`, commit: `875da16b7bc2a676d1a9d389bf22ee4579722073`.
  
  ## Test system
  * Ubuntu 20.04, with Docker https://hub.docker.com/repository/docker/thebloke/runpod-pytorch-new/general
  * CUDA toolkit 11.6
  * NVidia 4090 24GB
  * [WizardLM 7B 128g](https://huggingface.co/TheBloke/wizardLM-7B-GPTQ)
  
  ## Test method
  * All benchmarking done in text-gen-ui, using the output time and token/s reported by it.
  * text-gen-ui restarted between each test.
  * Output limit = 512 tokens.
  * 'Default' paramater set used + 'ban eos token' set so as to always get 512 tokens returned
  * Run one test and discard results as warm up, then record 4 results.
  
  ## Results spreadsheet and overview charts
  
  ### Spreadsheet
  [Google sheets with results and charts](https://docs.google.com/spreadsheets/d/1_VWHD-5jgVq5_nlP_0KJeKLsWDkZrZNBFdGR7YRMuME/edit?usp=sharing)
  
  ### Charts
  
  ![benchmark no-stream, model: no-act-order](https://docs.google.com/spreadsheets/d/e/2PACX-1vSQBua83h9f0-8IjSlCVQaKO7ivVmZeH7FiI_hqGPCntwvDfh525dzuXVdgoddSR2lQ-DBPYxHXEZ3u/pubchart?oid=637326428&format=image)
  
  ![benchmark no-stream, model: act-order](https://docs.google.com/spreadsheets/d/e/2PACX-1vSQBua83h9f0-8IjSlCVQaKO7ivVmZeH7FiI_hqGPCntwvDfh525dzuXVdgoddSR2lQ-DBPYxHXEZ3u/pubchart?oid=1267765257&format=image)
  
  A chart showing streaming performance is in the spreadsheet.
  
  ## Description of results
  
  ### AutoGPTQ vs 'ooba' CUDA with --no-stream
  * AutoGPTQ CUDA outperforms GfL 'ooba' CUDA by 15% on a no-act-order model
  * AutoGPTQ CUDA outperforms GfL 'ooba' CUDA by 10% on an act-order model (comparing AutoGPTQ on act-order model to GfL on no-act-order model)
  * AutoGPTQ Triton is 5% slower than GfL 'ooba' CUDA
  * AutoGPTQ Triton is 20% slower than AutoGPTQ CUDA
  
  ### AutoGPTQ vs 'ooba' CUDA with streaming (no-act-order model)
  * AutoGPTQ CUDA outperforms GfL 'ooba' CUDA by 12%
  * AutoGPTQ CUDA outperforms AutoGPTQ Triton by 13%
  * AutoGPTQ Triton outperforms GfL 'ooba' CUDA by 15%
    * Interesting that with streaming on, Triton does better than GfL CUDA. 
  
  ### desc_act models vs non-desc_act models
  * AutoGPTQ CUDA is 4.5% slower on a desc_act model vs not
  * AutoGPTQ Triton has no performance difference between desc_act model vs not
  * AutoGPTQ CUDA records significantly higher GPU usage % on desc_act models
    * 80% usage with desc_act + fused_attn vs 30% with no-desc_act model
    * This might be a problem on weaker cards? That needs testing.
  * AutoGPTQ Triton has only a few % extra GPU usage with desc_act models.
  
  ### fused_attn and fused_mlp
  * AutoGPTQ CUDA: fused_attn increases performance by 20%
    * This seems to account for nearly all the performance difference between AutoGPTQ CUDA and ooba GfL CUDA
  * AutoGPTQ Triton: fused_mlp on its own increases performance by 15% 
  * AutoGPTQ Triton: fused_attn on its own increases performance by 26%
  * AutoGPTQ Triton: fused_mlp and fused_attn together increases performance over no/no by 48%
  
  ### Slow loading time with AutoGPTQ Triton
  * AutoGPTQ Triton takes *significantly* longer to load a model vs CUDA
    * I didn't record yet benchmarks for this, but from looking through my logs I see:
      * CUDA: 2 -3 seconds
      * Triton:  40-45 seconds (with or without fused_mlp)
      * Triton + fused_attn: 55 - 90 seconds
    * I'll add this to the benchmark table later. 
   
  ## Results table
  
  |Implementation|Method|Streaming|Model type  |fused_attn|fused_mlp|GPU usage max %|VRAM max after 512 tok|Avg token/s|Run 1 tokens/s|Run 2 tokens/s|Run 3 tokens/s|Run 4 tokens/s|
  |--------------|------|---------|------------|----------|---------|---------------|----------------------|-----------|--------------|--------------|--------------|--------------|
  |ooba GfL      |CUDA  |No       |no-act-order|N/A       |N/A      |25%            |5837                  |23.84      |23.86         |23.91         |23.70         |23.90         |
  |AutoGPTQ      |CUDA  |No       |no-act-order|No        |N/A      |24%            |6711                  |22.66      |22.63         |22.63         |22.78         |22.61         |
  |AutoGPTQ      |CUDA  |No       |no-act-order|Yes       |N/A      |28%            |6849                  |27.22      |27.23         |27.33         |27.33         |27.00         |
  |AutoGPTQ      |Triton|No       |no-act-order|No        |No       |27%            |6055                  |15.25      |15.25         |15.25         |15.29         |15.20         |
  |AutoGPTQ      |Triton|No       |no-act-order|No        |Yes      |30%            |6691                  |17.48      |17.52         |17.51         |17.43         |17.47         |
  |AutoGPTQ      |Triton|No       |no-act-order|Yes       |No       |30%            |6013                  |19.33      |19.37         |19.42         |19.29         |19.24         |
  |AutoGPTQ      |Triton|No       |no-act-order|Yes       |Yes      |34%            |6649                  |22.58      |22.19         |22.67         |22.70         |22.75         |
  |AutoGPTQ      |CUDA  |No       |act-order   |No        |N/A      |64%            |6059                  |20.35      |20.38         |20.42         |20.31         |20.30         |
  |AutoGPTQ      |CUDA  |No       |act-order   |Yes       |N/A      |80%            |6079                  |26.02      |26.12         |26.15         |26.18         |25.61         |
  |AutoGPTQ      |Triton|No       |act-order   |No        |No       |30%            |6057                  |15.39      |15.47         |15.30         |15.35         |15.42         |
  |AutoGPTQ      |Triton|No       |act-order   |No        |yes      |33%            |6691                  |17.48      |17.54         |17.53         |17.38         |17.48         |
  |AutoGPTQ      |Triton|No       |act-order   |Yes       |No       |33%            |6013                  |19.55      |19.56         |19.59         |19.51         |19.55         |
  |AutoGPTQ      |Triton|No       |act-order   |Yes       |yes      |38%            |6649                  |22.86      |22.86         |23.01         |22.98         |22.57         |
  |ooba GfL      |CUDA  |Yes      |no-act-order|N/A       |N/A      |17%            |5837                  |14.93      |14.86         |14.77         |15.05         |15.05         |
  |AutoGPTQ      |CUDA  |Yes      |no-act-order|No        |N/A      |20%            |6711                  |16.85      |16.94         |16.84         |16.87         |16.76         |
  |AutoGPTQ      |CUDA  |Yes      |no-act-order|Yes       |no       |22%            |6849                  |19.55      |19.87         |19.40         |19.50         |19.41         |
  |AutoGPTQ      |Triton|Yes      |no-act-order|Yes       |Yes      |27%            |6429                  |17.19      |17.08         |17.38         |17.15         |17.14         |
  |AutoGPTQ      |Triton|Yes      |act-order   |No        |No       |25%            |6055                  |12.43      |12.39         |12.46         |12.40         |12.47         |
  |AutoGPTQ      |Triton|Yes      |act-order   |Yes       |Yes      |33%            |6649                  |17.19      |17.03         |17.03         |17.26         |17.42         |
  
  ## Benchmark logs
  
  <details>
    <summary>Benchmark logs in full</summary>
  
  ### ooba GPTQ-for-LLaMA CUDA no streaming (`--no-stream`). no-act-order model. no fused_attn
  
  #### Command
  ```
  python server.py --model wiz-no-act --wbits 4 --groupsize 128 --model_type llama  --listen --no-stream
  ```
  #### Benchmark
  GPU usage max: 25%, VRAM idle: 6037, VRAM after 512 tokens: 5837
  ```
  Output generated in 21.46 seconds (23.86 tokens/s, 512 tokens, context 16, seed 104167586)
  Output generated in 21.41 seconds (23.91 tokens/s, 512 tokens, context 16, seed 448558865)
  Output generated in 21.60 seconds (23.70 tokens/s, 512 tokens, context 16, seed 816202521)
  Output generated in 21.42 seconds (23.90 tokens/s, 512 tokens, context 16, seed 63649370)
  ```
  
  ### ooba GPTQ-for-LLaMA CUDA with streaming. no-act-order model. no fused_attn
  
  #### Command
  ```
  python server.py --model wiz-no-act --wbits 4 --groupsize 128 --model_type llama  --listen
  ```
  #### Benchmark
  GPU usage max: 17%, VRAM idle: 5247, VRAM after 512 tokens: 5837
  ```
  Output generated in 34.39 seconds (14.86 tokens/s, 511 tokens, context 16, seed 572742302)
  Output generated in 34.60 seconds (14.77 tokens/s, 511 tokens, context 16, seed 677465334)
  Output generated in 33.95 seconds (15.05 tokens/s, 511 tokens, context 16, seed 1685629937)
  Output generated in 33.95 seconds (15.05 tokens/s, 511 tokens, context 16, seed 1445023832)
  ```
  ### AutoGPTQ CUDA no streaming (`--no-stream`). no-act-order model. fused_attn enabled
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq  --listen --quant_attn --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 28%, VRAM idle: 6849, VRAM after 512 tokens: 6849
  ```
  Output generated in 18.81 seconds (27.23 tokens/s, 512 tokens, context 16, seed 1130150188)
  Output generated in 18.74 seconds (27.33 tokens/s, 512 tokens, context 16, seed 939013757)
  Output generated in 18.73 seconds (27.33 tokens/s, 512 tokens, context 16, seed 1724107769)
  Output generated in 18.97 seconds (27.00 tokens/s, 512 tokens, context 16, seed 54252597)
  ```
  
  ### AutoGPTQ CUDA with streaming. no-act-order model. fused_attn enabled
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq  --listen --quant_attn --wbits 4 --groupsize 128 --model_type llama 
  ```
  #### Benchmark
  
  GPU usage max: 22%, VRAM idle: 5437, VRAM after 512 tokens: 6849
  ```
  Output generated in 25.71 seconds (19.87 tokens/s, 511 tokens, context 16, seed 1472734050)
  Output generated in 26.33 seconds (19.40 tokens/s, 511 tokens, context 16, seed 1285036592)
  Output generated in 26.20 seconds (19.50 tokens/s, 511 tokens, context 16, seed 938935319)
  Output generated in 26.32 seconds (19.41 tokens/s, 511 tokens, context 16, seed 2142008394)
  ```
  
  ###  AutoGPTQ CUDA no streaming (`--no-stream`). no-act-order model. no fused_attn 
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 24%, VRAM idle: 6711, VRAM after 512 tokens: 6711
  ```
  Output generated in 22.63 seconds (22.63 tokens/s, 512 tokens, context 16, seed 1551481428)
  Output generated in 22.63 seconds (22.63 tokens/s, 512 tokens, context 16, seed 1993869704)
  Output generated in 22.48 seconds (22.78 tokens/s, 512 tokens, context 16, seed 596462747)
  Output generated in 22.64 seconds (22.61 tokens/s, 512 tokens, context 16, seed 619504695)
  ```
  
  ### AutoGPTQ CUDA with streaming. no-act-order model. no fused_attn 
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq  --listen  --wbits 4 --groupsize 128 --model_type llama 
  ```
  #### Benchmark
  
  GPU usage max: 20%, VRAM idle: 5277, VRAM after 512 tokens: 6711
  ```
  Output generated in 30.16 seconds (16.94 tokens/s, 511 tokens, context 16, seed 709588940)
  Output generated in 30.34 seconds (16.84 tokens/s, 511 tokens, context 16, seed 574596607)
  Output generated in 30.30 seconds (16.87 tokens/s, 511 tokens, context 16, seed 16071815)
  Output generated in 30.48 seconds (16.76 tokens/s, 511 tokens, context 16, seed 1202346043)
  ```
  
  ### AutoGPTQ CUDA no streaming (`--no-stream`). act-order / desc_act model. fused_attn=yes
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq  --listen --quant_attn --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 80%, VRAM idle: 6077, VRAM after 512 tokens: 6079
  ```
  Output generated in 19.60 seconds (26.12 tokens/s, 512 tokens, context 16, seed 1857860293)
  Output generated in 19.58 seconds (26.15 tokens/s, 512 tokens, context 16, seed 616647949)
  Output generated in 19.56 seconds (26.18 tokens/s, 512 tokens, context 16, seed 1384039801)
  Output generated in 19.99 seconds (25.61 tokens/s, 512 tokens, context 16, seed 411623614)
  ```
  
  ### AutoGPTQ CUDA no streaming (`--no-stream`). act-order / desc_act model.  fused_attn=no
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq  --listen --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 64%, VRAM idle: 6059, VRAM after 512 tokens: 6059
  ```
  Output generated in 25.12 seconds (20.38 tokens/s, 512 tokens, context 16, seed 1777836493)
  Output generated in 25.07 seconds (20.42 tokens/s, 512 tokens, context 16, seed 349075793)
  Output generated in 25.21 seconds (20.31 tokens/s, 512 tokens, context 16, seed 188931785)
  Output generated in 25.22 seconds (20.30 tokens/s, 512 tokens, context 16, seed 485419750)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). no-act-order model. fused_attn=yes. fused_mlp=yes
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq --autogptq-triton --fused_mlp --quant_attn  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 34%, VRAM idle: 6649, VRAM after 512 tokens: 6649
  ```
  Output generated in 23.07 seconds (22.19 tokens/s, 512 tokens, context 16, seed 1396024982)
  Output generated in 22.59 seconds (22.67 tokens/s, 512 tokens, context 16, seed 1322798716)
  Output generated in 22.56 seconds (22.70 tokens/s, 512 tokens, context 16, seed 935785726)
  Output generated in 22.50 seconds (22.75 tokens/s, 512 tokens, context 16, seed 2135223819)
  ```
  
  ### AutoGPTQ Triton with streaming. no-act-order model. fused_attn=yes. fused_mlp=yes
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq --autogptq-triton --fused_mlp --quant_attn  --listen  --wbits 4 --groupsize 128 --model_type llama 
  ```
  #### Benchmark
  
  GPU usage max: 27%, VRAM idle: 6299, VRAM after 512 tokens: 6429
  ```
  Output generated in 29.92 seconds (17.08 tokens/s, 511 tokens, context 16, seed 1687853126)
  Output generated in 29.40 seconds (17.38 tokens/s, 511 tokens, context 16, seed 1796675019)
  Output generated in 29.79 seconds (17.15 tokens/s, 511 tokens, context 16, seed 1342449921)
  Output generated in 29.81 seconds (17.14 tokens/s, 511 tokens, context 16, seed 1283884954)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). no-act-order model. fused_attn=no. fused_mlp=no
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq --autogptq-triton  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 27%, VRAM idle: 6055, VRAM after 512 tokens: 6055
  ```
  Output generated in 33.57 seconds (15.25 tokens/s, 512 tokens, context 16, seed 1071469137)
  Output generated in 33.58 seconds (15.25 tokens/s, 512 tokens, context 16, seed 1554707022)
  Output generated in 33.48 seconds (15.29 tokens/s, 512 tokens, context 16, seed 588803760)
  Output generated in 33.69 seconds (15.20 tokens/s, 512 tokens, context 16, seed 719688473)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). no-act-order model. fused_attn=no. fused_mlp=yes
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq --autogptq-triton --fused_mlp  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 30%, VRAM idle: 6691, VRAM after 512 tokens: 6691
  ```
  Output generated in 29.23 seconds (17.52 tokens/s, 512 tokens, context 16, seed 1413673599)
  Output generated in 29.24 seconds (17.51 tokens/s, 512 tokens, context 16, seed 2120666307)
  Output generated in 29.38 seconds (17.43 tokens/s, 512 tokens, context 16, seed 2057265550)
  Output generated in 29.32 seconds (17.47 tokens/s, 512 tokens, context 16, seed 1082953773)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). no-act-order model. fused_attn=yes. fused_mlp=no
  
  #### Command
  ```
  python server.py --model wiz-no-act  --autogptq --autogptq-triton --quant_attn  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 30%, VRAM idle: 6013, VRAM after 512 tokens: 6013
  ```
  Output generated in 26.43 seconds (19.37 tokens/s, 512 tokens, context 16, seed 1512231234)
  Output generated in 26.36 seconds (19.42 tokens/s, 512 tokens, context 16, seed 2018026458)
  Output generated in 26.54 seconds (19.29 tokens/s, 512 tokens, context 16, seed 1882161798)
  Output generated in 26.61 seconds (19.24 tokens/s, 512 tokens, context 16, seed 1512440780)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). act-order/desc_act model. fused_attn=yes. fused_mlp=yes
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq --autogptq-triton --fused_mlp --quant_attn  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max: 38%, VRAM idle: 6649, VRAM after 512 tokens: 6649
  ```
  Output generated in 22.40 seconds (22.86 tokens/s, 512 tokens, context 16, seed 1359206825)
  Output generated in 22.25 seconds (23.01 tokens/s, 512 tokens, context 16, seed 609149608)
  Output generated in 22.28 seconds (22.98 tokens/s, 512 tokens, context 16, seed 226374340)
  Output generated in 22.68 seconds (22.57 tokens/s, 512 tokens, context 16, seed 1070157383)
  ```
  
  ### AutoGPTQ Triton with streaming. act-order/desc_act model. fused_attn=yes. fused_mlp=yes
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq --autogptq-triton --fused_mlp --quant_attn  --listen  --wbits 4 --groupsize 128 --model_type llama 
  ```
  #### Benchmark
  
  GPU usage max: 33%, VRAM idle: 6299, VRAM after 512 tokens: 6649
  ```
  Output generated in 30.00 seconds (17.03 tokens/s, 511 tokens, context 16, seed 456349974)
  Output generated in 30.00 seconds (17.03 tokens/s, 511 tokens, context 16, seed 767092960)
  Output generated in 29.61 seconds (17.26 tokens/s, 511 tokens, context 16, seed 381684718)
  Output generated in 29.33 seconds (17.42 tokens/s, 511 tokens, context 16, seed 283294303)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). act-order/desc_act model. fused_attn=no. fused_mlp=yes
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq --autogptq-triton --fused_mlp  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max:33%, VRAM idle: 6691, VRAM after 512 tokens: 6691
  ```
  Output generated in 29.19 seconds (17.54 tokens/s, 512 tokens, context 16, seed 1575265983)
  Output generated in 29.21 seconds (17.53 tokens/s, 512 tokens, context 16, seed 1616043283)
  Output generated in 29.47 seconds (17.38 tokens/s, 512 tokens, context 16, seed 1647334679)
  Output generated in 29.29 seconds (17.48 tokens/s, 512 tokens, context 16, seed 256676128)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). act-order/desc_act model. fused_attn=yes. fused_mlp=no
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq --autogptq-triton --quant_attn  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max:33%, VRAM idle: 6013, VRAM after 512 tokens: 6013
  ```
  Output generated in 26.18 seconds (19.56 tokens/s, 512 tokens, context 16, seed 289490511)
  Output generated in 26.13 seconds (19.59 tokens/s, 512 tokens, context 16, seed 2123553925)
  Output generated in 26.24 seconds (19.51 tokens/s, 512 tokens, context 16, seed 563248868)
  Output generated in 26.19 seconds (19.55 tokens/s, 512 tokens, context 16, seed 1773520422)
  ```
  
  ### AutoGPTQ Triton no streaming (`--no-stream`). act-order/desc_act model. fused_attn=no. fused_mlp=no
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq --autogptq-triton  --listen  --wbits 4 --groupsize 128 --model_type llama --no-stream
  ```
  #### Benchmark
  
  GPU usage max:30%, VRAM idle: 6057, VRAM after 512 tokens: 6057
  ```
  Output generated in 33.09 seconds (15.47 tokens/s, 512 tokens, context 16, seed 1881763981)
  Output generated in 33.47 seconds (15.30 tokens/s, 512 tokens, context 16, seed 83555537)
  Output generated in 33.36 seconds (15.35 tokens/s, 512 tokens, context 16, seed 332008224)
  Output generated in 33.20 seconds (15.42 tokens/s, 512 tokens, context 16, seed 657280485)
  ```
  
  ### AutoGPTQ Triton with streaming. act-order/desc_act model. fused_attn=no. fused_mlp=no
  
  #### Command
  ```
  python server.py --model wiz-act  --autogptq --autogptq-triton  --listen  --wbits 4 --groupsize 128 --model_type llama
  ```
  #### Benchmark
  
  GPU usage max: 25%, VRAM idle: 5503, VRAM after 512 tokens: 6055
  ```
  Output generated in 41.23 seconds (12.39 tokens/s, 511 tokens, context 16, seed 1164743843)
  Output generated in 41.02 seconds (12.46 tokens/s, 511 tokens, context 16, seed 509370735)
  Output generated in 41.21 seconds (12.40 tokens/s, 511 tokens, context 16, seed 246113358)
  Output generated in 40.99 seconds (12.47 tokens/s, 511 tokens, context 16, seed 667851869)
  ```
  </details>