c4ai-command-r-plus, quantized to GPTQ with these parameters:
python3 quant.py aCohereForAI/c4ai-command-r-plus /workspace/commandplus custom --bits 4 --group_size 128 --desc_act 1 --damp 0.1 --dtype float16 --seqlen 16384 --num_samples 256 --cache_examples 0 --trust_remote_code
The dataset used was openerotica/erotiquant2. I have included a script reconstitute.py to merge the files into one. Depending on the backend you might need to delete the index file after the files have been merged. I'll try to do this all in a better way once I work after I test out how marlin stacks up to exl2 for this model.
- Downloads last month
- 73
This model does not have enough activity to be deployed to Inference API (serverless) yet.
Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.