metadata

license: apache-2.0
tags:
  - function-calling

Fireworks Function Calling (FireFunction) Model V2

FireFunction is a state-of-the-art function calling model with a commercially viable license. Key info and highlights:

🐾 Successor of the FireFunction model

📏 Signifficant quality improvements over FireFunction v1 across the broad range of metrics

🔆 Support of parallel function calling (unlike FireFunction v1) and good instruction following

💡 Hosted on the Fireworks platform

Intended Use and Limitations

Supported usecases

The model was tuned to perfom well on a range of usecases including:

general instruction following
multi-turn chat mixing vanilla messages with function calls
single- and parallel function calling
up to 20 function specs supported at once
structured information extraction

Out-of-Scope Use

The model was not optimized for the following use cases:

100+ function specs
nested function calling

Metrics

Benchmark	Firefunction v1	Firefunction v2	Llama 3 70b Instruct	Gpt-4o
Gorilla simple	0.91	0.94	0.925	0.88
Gorilla multiple_function	0.92	0.91	0.86	0.91
Gorilla parallel_function	0	0.9	0.86	0.89
Gorilla parallel_multiple_function	0	0.8	0.615	0.72
Nexus parallel	0.38	0.53	0.3	0.47
Mtbench	0.73	0.84	0.89	0.93
Average	0.49	0.82	0.74	0.8

Example Usage

See documentation for more detail.

from transformers import AutoModelForCausalLM, AutoTokenizer
import json
from datetime import datetime

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("fireworks-ai/firefunction-v2", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("fireworks-ai/firefunction-v2")

function_spec = [
    {
        "name": "get_stock_price",
        "description": "Get the current stock price",
        "parameters": {
            "type": "object",
            "properties": {
                "symbol": {
                    "type": "string",
                    "description": "The stock symbol, e.g. AAPL, GOOG"
                }
            },
            "required": [
                "symbol"
            ]
        }
    },
    {
        "name": "check_word_anagram",
        "description": "Check if two words are anagrams of each other",
        "parameters": {
            "type": "object",
            "properties": {
                "word1": {
                    "type": "string",
                    "description": "The first word"
                },
                "word2": {
                    "type": "string",
                    "description": "The second word"
                }
            },
            "required": [
                "word1",
                "word2"
            ]
        }
    }
]
functions = json.dumps(function_spec, indent=4)

messages = [
    {'role': 'system', 'content': 'You are a helpful assistant with access to functions. Use them if required.'},
    {'role': 'user', 'content': 'Hi, can you tell me the current stock price of google and netflix?'}
]

now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

model_inputs = tokenizer.apply_chat_template(messages, functions=functions, datetime=now, return_tensors="pt").to(model.device)

generated_ids = model.generate(model_inputs, max_new_tokens=128)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

fireworks-ai
/

llama-3-firefunction-v2