Edit model card

SkillTree Model Collection

Applying a skill to your model with SkillTree is akin to unlocking a new ability in a video game's skill tree. Just as you would enhance your character's capabilities by selecting and activating specific skills, you can augment your model's abilities by integrating specialized skills. Follow these steps to imbue your model with new prowess, enhancing its performance and versatility in a straightforward and intuitive manner.
Please note that SkillTree abilities may not function in all cases. To determine whether a specific skill is operational, refer to the Functionality Status.

What is SkillTree?

SkillTree represents a set of model weights derived from further pre-training or fine-tuning Large Language Models (LLMs) to extract specific capabilities, such as code generation or chatting abilities. These extracted "skills" can be combined with a specific LLM base model to enhance its capabilities. The concept is inspired by ChatVector, aiming to modularize and transfer distinct skills across models.

SkillTree Details

Uses

Limitation

  • Model Architecture: Llama2
  • Model Size: 6.74B
  • Compatible Models[optional]:

How to Apply Skill (Example)

# Import Library
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the target model to be applied skill
tokenizer = AutoTokenizer.from_pretrained(
    "tokyotech-llm/Swallow-7b-hf"
)
model = AutoModelForCausalLM.from_pretrained(
    "tokyotech-llm/Swallow-7b-hf",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load SkillTree
skill_tree = AutoModelForCausalLM.from_pretrained(
    "HachiML/SkillTree-llama2-7b-hf-Code",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Apply the skill to the target model
def apply_skill(model, skill_tree):
    # excluded object
    skip_layers = ["model.embed_tokens.weight", "model.norm.weight", "lm_head.weight"]
    # apply skill
    for k, v in model.state_dict().items():
        # layernorm is also excluded
        if (k in skip_layers) or ("layernorm" in k):
            continue
        vector = skill_tree.state_dict()[k]
        new_v = v + vector.to(v.device)
        v.copy_(new_v)
    return model

model = apply_skill(model, skill_tree)

# Push to hub
model_name = "HachiML/Swallow-7b-hf-CodeSkill"
tokenizer.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
model.save_pretrained(f"./models/{model_name}", repo_id=model_name, push_to_hub=True)
Downloads last month
13
Safetensors
Model size
6.74B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Collection including HachiML/SkillTree-Code-llama2-7b-hf