GGUF / IQ / Imatrix for Infinite-Sumika-9B
Why Importance Matrix?
Importance Matrix, at least based on my testing, has shown to improve the output and performance of "IQ"-type quantizations, where the compression becomes quite heavy. The Imatrix performs a calibration, using a provided dataset. Testing has shown that semi-randomized data can help perserve more important segments as the compression is applied.
Related discussions in Github: [1] [2]
The imatrix.txt file that I used contains general, semi-random data, with some added extra kink.
Infinite-Sumika-9B
This model is intended for fictional role-playing and storywriting.
It seems to be doing good with longer text, and prefers to write longer, storytelling-like responses sometimes.
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the passthrough merge method.
Models Merged
The following models were included in the merge:
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- model: Endevor/InfinityRP-v1-7B
layer_range: [0, 20]
- sources:
- model: localfultonextractor/Erosumika-7B-v2
layer_range: [12, 32]
merge_method: passthrough
dtype: float16
- Downloads last month
- 70
Model tree for ABX-AI/Infinite-Sumika-9B-GGUF-IQ-Imatrix
Base model
Endevor/InfinityRP-v1-7B