Thanks for the exl2! Can you do 4bpw?
#1
by
async0x42
- opened
I tried to do a quant for 4bpw to upload since it fits at full ctx for a 4090/4080, but it closes without messages at the 53rd step. If you saved the measurements and can generate that it would be great.