Initial GPTQ model commit

README.md

@@ -58,7 +58,7 @@ from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
 import argparse
 
 model_name_or_path = "TheBloke/samantha-1.1-llama-33B-GPTQ"
-model_basename = "gptq_model-4bit-128g"
+model_basename = "samantha-1.1-llama-33b-GPTQ-4bit--1g.act.order"
 
 use_triton = False
 
@@ -103,17 +103,17 @@ print(pipe(prompt_template)[0]['generated_text'])
 
 ## Provided files
 
-**gptq_model-4bit-128g.safetensors**
+**samantha-1.1-llama-33b-GPTQ-4bit--1g.act.order.safetensors**
 
 This will work with AutoGPTQ and CUDA versions of GPTQ-for-LLaMa. There are reports of issues with Triton mode of recent GPTQ-for-LLaMa. If you have issues, please use AutoGPTQ instead.
 
-It was created with group_size 128 to increase inference accuracy, but without --act-order (desc_act) to increase compatibility and improve inference speed.
+It was created without group_size to lower VRAM requirements, and with --act-order (desc_act) to boost inference accuracy as much as possible.
 
-* `gptq_model-4bit-128g.safetensors`
+* `samantha-1.1-llama-33b-GPTQ-4bit--1g.act.order.safetensors`
   * Works with AutoGPTQ in CUDA or Triton modes.
   * Works with GPTQ-for-LLaMa in CUDA mode.  May have issues with GPTQ-for-LLaMa Triton mode.
   * Works with text-generation-webui, including one-click-installers.
-  * Parameters: Groupsize = 128. Act Order / desc_act = False.
+  * Parameters: Groupsize = -1. Act Order / desc_act = True.
 
 <!-- footer start -->
 ## Discord