Superswallow-70b-RP-v0.3

Important Notice:

For personal and academic use only. Please check the description for details.

This model partially utilizes the parameters of Tulu V2 DPO finetuned based on Llama 2, so it may inherit the AI2 ImpACT license. Please use the model keeping in mind that there may be changes regarding the license if AI2 contacts me.

The AI2 ImpACT license includes information about data artifacts and model artifacts, but does not cover the case of directly applying parts of the LLM parameters of a model artifact to other models. However, I respect their research and great work, so I will change the license immediately if AI2 contacts me.

Description

This model is suitable for role-playing and storytelling.

This was created for personal and academic use only. This merge model uses only fine-tune models of Llama2, but some of the models used include those whose licenses for commercial use are unclear.

If there is a license problem, the rights holder should contact me directly. No license changes will be made due to contact from others.

Test environment

This model was tested using text-generation-webui. I use preset simple-1 and Null preset for Generation.

Recommendation

Use simple-1 settings:

• temperature: 0.7
• top_p: 0.9
• repetition_penalty: 1.15
• top_k: 20

Tested temperature Range

• temperature: 0.3 - 1.0

It works fine in most cases, but depending on the prompt, the output may become unstable at the temperature around 1.0.

If the output does not follow the user intent, please lower the temperature to 0.5 or less.

Tested repetition_penalty Range

• repetition_penalty: 1.0 - 1.15

It works fine in most cases, but depending on the prompt, the output may become unstable at the repetition_penalty around 1.0.

Prompt template

All prompt templates are available as well.

Tulu Style

<|user|>
Your message here!
<|assistant|>

For best results, format all inputs in this manner. Make sure to include a newline after <|assistant|>, this can affect generation quality quite a bit.

Swallow Style (Alpaca format)

以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。リクエストを適切に完了するための回答を記述してください。

### 指示:
{instruction}

### 応答:

Use the instruct model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "nitky/Superswallow-70b-RP-v0.3"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto", load_in_4bit = True)


PROMPT_DICT = {
    "prompt_input": (
        "以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。"
        "リクエストを適切に完了するための回答を記述してください。\n\n"
        "### 指示:\n{instruction}\n\n### 入力:\n{input}\n\n### 応答:"

    ),
    "prompt_no_input": (
        "以下に、あるタスクを説明する指示があります。"
        "リクエストを適切に完了するための回答を記述してください。\n\n"
        "### 指示:\n{instruction}\n\n### 応答:"
    ),
}

def create_prompt(instruction, input=None):
    """
    Generates a prompt based on the given instruction and an optional input.
    If input is provided, it uses the 'prompt_input' template from PROMPT_DICT.
    If no input is provided, it uses the 'prompt_no_input' template.

    Args:
        instruction (str): The instruction describing the task.
        input (str, optional): Additional input providing context for the task. Default is None.

    Returns:
        str: The generated prompt.
    """
    if input:
        # Use the 'prompt_input' template when additional input is provided
        return PROMPT_DICT["prompt_input"].format(instruction=instruction, input=input)
    else:
        # Use the 'prompt_no_input' template when no additional input is provided
        return PROMPT_DICT["prompt_no_input"].format(instruction=instruction)

# Example usage
instruction_example = "以下のトピックに関する詳細な情報を提供してください。"
input_example = "東京工業大学の主なキャンパスについて教えてください"
prompt = create_prompt(instruction_example, input_example)

input_ids = tokenizer.encode(
    prompt,
    add_special_tokens=False,
    return_tensors="pt"
)

tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    top_k=20,
    do_sample=True,
)

out = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(out)

Merge Details

Merge Method

This model was merged using the DARE TIES and the SLERP merge method using tokyotech-llm/Swallow-70b-instruct-hf as a base.

Models Merged

The following models were included in the merge:

• allenai/tulu-2-dpo-70b
• GOAT-AI/GOAT-70B-Storytelling
• dreamgen/opus-v0.5-70b
• Doctor-Shotgun/lzlv-limarpv3-l2-70b
• [LoRA] alac/Waxwing-Storytelling-70B-LoRA

Configuration

The command example:

# please change the path and options according to your environment
mergekit-mega --cuda Superswallow-70b-RP-v0.3.yml ~/text-generation-webui/models

The following YAML configuration was used to produce this model:

models:
  - model: nitky/Superswallow-70b-v0.3
    # no parameters necessary for base model
  - model: nitky/Swallow-70b-RP
    parameters:
      density: 1
      weight:
      - filter: mlp
        value: 0.1
      - filter: self_attn
        value: 0.25
      - value: 0 # fallback for rest of tensors.
merge_method: dare_ties
base_model: nitky/Superswallow-70b-v0.3
dtype: bfloat16
name: Superswallow-70b-RP-v0.3-base
---
models:
  - model: nitky/Superswallow-70b-v0.3
    # no parameters necessary for base model
  - model: nitky/Swallow-70b-RP
    parameters:
      density: 1
      weight:
      - filter: mlp
        value: [0.25, 0.1, 0.25, 0.1, 0.25, 0.1, 0.25, 0.1, 0.1]
      - filter: self_attn
        value: [0.25, 0.25, 0.1, 0.25, 0.1, 0.25, 0.1, 0.25, 0.25]
      - value: 0 # fallback for rest of tensors.
merge_method: dare_ties
base_model: nitky/Superswallow-70b-v0.3
dtype: bfloat16
name: Superswallow-70b-RP-v0.3-flavor
---
slices:
  - sources:
      - model: Superswallow-70b-RP-v0.3-base
        layer_range: [0, 80]
      - model: Superswallow-70b-RP-v0.3-flavor
        layer_range: [0, 80]
merge_method: slerp
base_model: Superswallow-70b-RP-v0.3-base
parameters:
  t: # model stabilization
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
dtype: bfloat16
name: Superswallow-70b-RP-v0.3